2023-11-18 11:50:15,602 INFO StreamThr :16687 [internal.py:wandb_internal():86] W&B internal server running at pid: 16687, started at: 2023-11-18 11:50:15.600888 2023-11-18 11:50:15,603 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: status 2023-11-18 11:50:15,607 INFO WriterThread:16687 [datastore.py:open_for_write():85] open: /mnt/e/wandb/run-20231118_115015-iyvqyo9v/run-iyvqyo9v.wandb 2023-11-18 11:50:15,609 DEBUG SenderThread:16687 [sender.py:send():380] send: header 2023-11-18 11:50:15,609 DEBUG SenderThread:16687 [sender.py:send():380] send: run 2023-11-18 11:50:16,119 INFO SenderThread:16687 [dir_watcher.py:__init__():211] watching files in: /mnt/e/wandb/run-20231118_115015-iyvqyo9v/files 2023-11-18 11:50:16,129 INFO SenderThread:16687 [sender.py:_start_run_threads():1124] run started: iyvqyo9v with start time 1700301015.601293 2023-11-18 11:50:16,131 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: check_version 2023-11-18 11:50:16,132 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: check_version 2023-11-18 11:50:16,177 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: run_start 2023-11-18 11:50:16,184 DEBUG HandlerThread:16687 [system_info.py:__init__():32] System info init 2023-11-18 11:50:16,184 DEBUG HandlerThread:16687 [system_info.py:__init__():47] System info init done 2023-11-18 11:50:16,184 INFO HandlerThread:16687 [system_monitor.py:start():194] Starting system monitor 2023-11-18 11:50:16,185 INFO SystemMonitor:16687 [system_monitor.py:_start():158] Starting system asset monitoring threads 2023-11-18 11:50:16,185 INFO HandlerThread:16687 [system_monitor.py:probe():214] Collecting system info 2023-11-18 11:50:16,186 INFO SystemMonitor:16687 [interfaces.py:start():190] Started cpu monitoring 2023-11-18 11:50:16,187 INFO SystemMonitor:16687 [interfaces.py:start():190] Started disk monitoring 2023-11-18 11:50:16,187 INFO SystemMonitor:16687 [interfaces.py:start():190] Started gpu monitoring 2023-11-18 11:50:16,188 INFO SystemMonitor:16687 [interfaces.py:start():190] Started memory monitoring 2023-11-18 11:50:16,189 INFO SystemMonitor:16687 [interfaces.py:start():190] Started network monitoring 2023-11-18 11:50:16,195 DEBUG HandlerThread:16687 [system_info.py:probe():196] Probing system 2023-11-18 11:50:16,294 DEBUG HandlerThread:16687 [gitlib.py:_init_repo():56] git repository is invalid 2023-11-18 11:50:16,294 DEBUG HandlerThread:16687 [system_info.py:probe():244] Probing system done 2023-11-18 11:50:16,294 DEBUG HandlerThread:16687 [system_monitor.py:probe():223] {'os': 'Linux-5.15.123.1-microsoft-standard-WSL2-x86_64-with-glibc2.35', 'python': '3.10.13', 'heartbeatAt': '2023-11-18T09:50:16.195837', 'startedAt': '2023-11-18T09:50:15.570899', 'docker': None, 'cuda': None, 'args': ('--deepspeed=ds_config.json', '--model_name_or_path=openai/whisper-large-v2', '--train_dataset_names', 'mozilla-foundation/common_voice_11_0', 'mozilla-foundation/common_voice_11_0', 'google/fleurs', 'Finnish-NLP/css10_audio_processed', 'Finnish-NLP/youtube_fi_110k_preprocessed', 'Finnish-NLP/voxpopuli_fi_audio_processed', 'Finnish-NLP/parliament_audio_processed', 'Finnish-NLP/aalto_eduskunta_asr_audio_processed', 'Finnish-NLP/fbc_dialog_audio_processed', 'Finnish-NLP/fbc_monolog_audio_processed', '--train_dataset_config_names', 'fi', 'fi', 'fi_fi', '', '', '', '', '', '', '', '--train_dataset_split_name', 'train', 'validation', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', '--train_dataset_text_column_names', 'sentence', 'sentence', 'raw_transcription', 'sentence', 'texts', 'sentence', 'sentence', 'sentence', 'sentence', 'sentence', '--eval_dataset_name=mozilla-foundation/common_voice_11_0', '--eval_dataset_config_name=fi', '--eval_split_name=test', '--language=finnish', '--model_index_name=Whisper Large V3 Finnish', '--max_steps=30000', '--output_dir=./', '--per_device_train_batch_size=2', '--per_device_eval_batch_size=2', '--gradient_accumulation_steps=2', '--logging_steps=25', '--learning_rate=3e-6', '--warmup_steps=1000', '--evaluation_strategy=steps', '--eval_steps=1000', '--save_strategy=steps', '--save_steps=1000', '--generation_max_length=225', '--length_column_name=input_length', '--max_duration_in_seconds=30', '--text_column_name=sentence', '--freeze_feature_encoder=False', '--report_to=wandb', '--metric_for_best_model=wer', '--greater_is_better=False', '--load_best_model_at_end', '--gradient_checkpointing', '--fp16', '--overwrite_output_dir', '--do_train', '--do_eval', '--predict_with_generate', '--do_normalize_eval', '--use_auth_token', '--push_to_hub'), 'state': 'running', 'program': '/mnt/e/run_speech_recognition_seq2seq_streaming.py', 'codePathLocal': 'run_speech_recognition_seq2seq_streaming.py', 'codePath': 'run_speech_recognition_seq2seq_streaming.py', 'host': 'DESKTOP-59O9VN1', 'username': 'rasmus', 'executable': '/home/rasmus/miniconda3/envs/WhisperFinetuneEnv/bin/python', 'cpu_count': 4, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3911.9949999999994, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 1006.853931427002, 'used': 37.27444076538086}}, 'gpu': 'NVIDIA GeForce RTX 4080', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 4080', 'memory_total': 17171480576}], 'memory': {'total': 54.92573928833008}} 2023-11-18 11:50:16,294 INFO HandlerThread:16687 [system_monitor.py:probe():224] Finished collecting system info 2023-11-18 11:50:16,294 INFO HandlerThread:16687 [system_monitor.py:probe():227] Publishing system info 2023-11-18 11:50:16,294 DEBUG HandlerThread:16687 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment 2023-11-18 11:50:16,296 DEBUG HandlerThread:16687 [system_info.py:_save_pip():68] Saving pip packages done 2023-11-18 11:50:16,297 DEBUG HandlerThread:16687 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment 2023-11-18 11:50:17,122 INFO Thread-12 :16687 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_115015-iyvqyo9v/files/conda-environment.yaml 2023-11-18 11:50:17,123 INFO Thread-12 :16687 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_115015-iyvqyo9v/files/requirements.txt 2023-11-18 11:50:18,127 INFO Thread-12 :16687 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_115015-iyvqyo9v/files/conda-environment.yaml 2023-11-18 11:50:18,204 DEBUG HandlerThread:16687 [system_info.py:_save_conda():87] Saving conda packages done 2023-11-18 11:50:18,216 INFO HandlerThread:16687 [system_monitor.py:probe():229] Finished publishing system info 2023-11-18 11:50:18,219 DEBUG SenderThread:16687 [sender.py:send():380] send: files 2023-11-18 11:50:18,219 INFO SenderThread:16687 [sender.py:_save_file():1380] saving file wandb-metadata.json with policy now 2023-11-18 11:50:18,225 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 11:50:18,226 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 11:50:18,226 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: stop_status 2023-11-18 11:50:18,403 DEBUG SenderThread:16687 [sender.py:send():380] send: telemetry 2023-11-18 11:50:18,404 DEBUG SenderThread:16687 [sender.py:send():380] send: config 2023-11-18 11:50:18,404 DEBUG SenderThread:16687 [sender.py:send():380] send: metric 2023-11-18 11:50:18,405 DEBUG SenderThread:16687 [sender.py:send():380] send: telemetry 2023-11-18 11:50:18,405 DEBUG SenderThread:16687 [sender.py:send():380] send: metric 2023-11-18 11:50:18,406 WARNING SenderThread:16687 [sender.py:send_metric():1331] Seen metric with glob (shouldn't happen) 2023-11-18 11:50:18,406 DEBUG SenderThread:16687 [sender.py:send():380] send: telemetry 2023-11-18 11:50:18,813 INFO wandb-upload_0:16687 [upload_job.py:push():131] Uploaded file /tmp/tmps37fqhtowandb/tuibr4vd-wandb-metadata.json 2023-11-18 11:50:19,132 INFO Thread-12 :16687 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_115015-iyvqyo9v/files/output.log 2023-11-18 11:50:19,133 INFO Thread-12 :16687 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_115015-iyvqyo9v/files/wandb-metadata.json 2023-11-18 11:50:21,141 INFO Thread-12 :16687 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_115015-iyvqyo9v/files/output.log 2023-11-18 11:50:21,598 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:50:23,148 INFO Thread-12 :16687 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_115015-iyvqyo9v/files/output.log 2023-11-18 11:50:26,599 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:50:31,600 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:50:33,225 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 11:50:33,225 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: stop_status 2023-11-18 11:50:33,227 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 11:50:37,473 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:50:42,474 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:50:47,477 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:50:48,225 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 11:50:48,226 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: stop_status 2023-11-18 11:50:48,248 INFO Thread-12 :16687 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_115015-iyvqyo9v/files/config.yaml 2023-11-18 11:50:48,275 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 11:50:53,448 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:50:58,449 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:51:03,225 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 11:51:03,226 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: stop_status 2023-11-18 11:51:03,275 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 11:51:03,468 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:51:08,469 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:51:13,470 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:51:16,190 DEBUG SystemMonitor:16687 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2023-11-18 11:51:16,191 DEBUG SenderThread:16687 [sender.py:send():380] send: stats 2023-11-18 11:51:18,225 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 11:51:18,226 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: stop_status 2023-11-18 11:51:18,265 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 11:51:19,468 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:51:24,469 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:51:29,470 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:51:33,225 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 11:51:33,226 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: stop_status 2023-11-18 11:51:33,275 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 11:51:35,459 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:51:40,460 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:51:45,461 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:51:46,192 DEBUG SenderThread:16687 [sender.py:send():380] send: stats 2023-11-18 11:51:48,226 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 11:51:48,226 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: stop_status 2023-11-18 11:51:48,275 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 11:51:50,476 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:51:55,477 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:52:00,478 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:52:03,225 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 11:52:03,226 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: stop_status 2023-11-18 11:52:03,275 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 11:52:06,429 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:52:11,430 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:52:14,578 INFO Thread-12 :16687 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_115015-iyvqyo9v/files/output.log 2023-11-18 11:52:16,192 DEBUG SenderThread:16687 [sender.py:send():380] send: stats 2023-11-18 11:52:16,586 INFO Thread-12 :16687 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_115015-iyvqyo9v/files/output.log 2023-11-18 11:52:17,129 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:52:17,133 DEBUG SenderThread:16687 [sender.py:send():380] send: exit 2023-11-18 11:52:17,133 INFO SenderThread:16687 [sender.py:send_exit():585] handling exit code: 1 2023-11-18 11:52:17,133 INFO SenderThread:16687 [sender.py:send_exit():587] handling runtime: 120 2023-11-18 11:52:17,144 INFO SenderThread:16687 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 11:52:17,145 INFO SenderThread:16687 [sender.py:send_exit():593] send defer 2023-11-18 11:52:17,146 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:52:17,146 INFO HandlerThread:16687 [handler.py:handle_request_defer():172] handle defer: 0 2023-11-18 11:52:17,146 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: defer 2023-11-18 11:52:17,146 INFO SenderThread:16687 [sender.py:send_request_defer():609] handle sender defer: 0 2023-11-18 11:52:17,147 INFO SenderThread:16687 [sender.py:transition_state():613] send defer: 1 2023-11-18 11:52:17,147 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:52:17,147 INFO HandlerThread:16687 [handler.py:handle_request_defer():172] handle defer: 1 2023-11-18 11:52:17,147 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: defer 2023-11-18 11:52:17,147 INFO SenderThread:16687 [sender.py:send_request_defer():609] handle sender defer: 1 2023-11-18 11:52:17,148 INFO SenderThread:16687 [sender.py:transition_state():613] send defer: 2 2023-11-18 11:52:17,148 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:52:17,148 INFO HandlerThread:16687 [handler.py:handle_request_defer():172] handle defer: 2 2023-11-18 11:52:17,148 INFO HandlerThread:16687 [system_monitor.py:finish():203] Stopping system monitor 2023-11-18 11:52:17,149 DEBUG SystemMonitor:16687 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2023-11-18 11:52:17,149 INFO HandlerThread:16687 [interfaces.py:finish():202] Joined cpu monitor 2023-11-18 11:52:17,149 DEBUG SystemMonitor:16687 [system_monitor.py:_start():183] Publishing last batch of metrics 2023-11-18 11:52:17,150 INFO HandlerThread:16687 [interfaces.py:finish():202] Joined disk monitor 2023-11-18 11:52:17,158 INFO HandlerThread:16687 [interfaces.py:finish():202] Joined gpu monitor 2023-11-18 11:52:17,158 INFO HandlerThread:16687 [interfaces.py:finish():202] Joined memory monitor 2023-11-18 11:52:17,158 INFO HandlerThread:16687 [interfaces.py:finish():202] Joined network monitor 2023-11-18 11:52:17,159 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: defer 2023-11-18 11:52:17,159 INFO SenderThread:16687 [sender.py:send_request_defer():609] handle sender defer: 2 2023-11-18 11:52:17,160 INFO SenderThread:16687 [sender.py:transition_state():613] send defer: 3 2023-11-18 11:52:17,160 DEBUG SenderThread:16687 [sender.py:send():380] send: stats 2023-11-18 11:52:17,160 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:52:17,160 INFO HandlerThread:16687 [handler.py:handle_request_defer():172] handle defer: 3 2023-11-18 11:52:17,161 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: defer 2023-11-18 11:52:17,161 INFO SenderThread:16687 [sender.py:send_request_defer():609] handle sender defer: 3 2023-11-18 11:52:17,161 INFO SenderThread:16687 [sender.py:transition_state():613] send defer: 4 2023-11-18 11:52:17,161 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:52:17,161 INFO HandlerThread:16687 [handler.py:handle_request_defer():172] handle defer: 4 2023-11-18 11:52:17,162 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: defer 2023-11-18 11:52:17,162 INFO SenderThread:16687 [sender.py:send_request_defer():609] handle sender defer: 4 2023-11-18 11:52:17,162 INFO SenderThread:16687 [sender.py:transition_state():613] send defer: 5 2023-11-18 11:52:17,162 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:52:17,162 INFO HandlerThread:16687 [handler.py:handle_request_defer():172] handle defer: 5 2023-11-18 11:52:17,163 DEBUG SenderThread:16687 [sender.py:send():380] send: summary 2023-11-18 11:52:17,164 INFO SenderThread:16687 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 11:52:17,165 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: defer 2023-11-18 11:52:17,165 INFO SenderThread:16687 [sender.py:send_request_defer():609] handle sender defer: 5 2023-11-18 11:52:17,166 INFO SenderThread:16687 [sender.py:transition_state():613] send defer: 6 2023-11-18 11:52:17,166 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:52:17,166 INFO HandlerThread:16687 [handler.py:handle_request_defer():172] handle defer: 6 2023-11-18 11:52:17,166 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: defer 2023-11-18 11:52:17,167 INFO SenderThread:16687 [sender.py:send_request_defer():609] handle sender defer: 6 2023-11-18 11:52:17,167 INFO SenderThread:16687 [sender.py:transition_state():613] send defer: 7 2023-11-18 11:52:17,167 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:52:17,167 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:52:17,167 INFO HandlerThread:16687 [handler.py:handle_request_defer():172] handle defer: 7 2023-11-18 11:52:17,168 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: defer 2023-11-18 11:52:17,168 INFO SenderThread:16687 [sender.py:send_request_defer():609] handle sender defer: 7 2023-11-18 11:52:17,590 INFO Thread-12 :16687 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_115015-iyvqyo9v/files/wandb-summary.json 2023-11-18 11:52:18,132 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: poll_exit 2023-11-18 11:52:18,555 INFO SenderThread:16687 [sender.py:transition_state():613] send defer: 8 2023-11-18 11:52:18,555 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: poll_exit 2023-11-18 11:52:18,555 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:52:18,556 INFO HandlerThread:16687 [handler.py:handle_request_defer():172] handle defer: 8 2023-11-18 11:52:18,557 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: defer 2023-11-18 11:52:18,557 INFO SenderThread:16687 [sender.py:send_request_defer():609] handle sender defer: 8 2023-11-18 11:52:18,557 INFO SenderThread:16687 [job_builder.py:build():298] Attempting to build job artifact 2023-11-18 11:52:18,563 INFO SenderThread:16687 [job_builder.py:_get_source_type():439] no source found 2023-11-18 11:52:18,563 INFO SenderThread:16687 [sender.py:transition_state():613] send defer: 9 2023-11-18 11:52:18,563 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:52:18,564 INFO HandlerThread:16687 [handler.py:handle_request_defer():172] handle defer: 9 2023-11-18 11:52:18,564 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: defer 2023-11-18 11:52:18,564 INFO SenderThread:16687 [sender.py:send_request_defer():609] handle sender defer: 9 2023-11-18 11:52:18,564 INFO SenderThread:16687 [dir_watcher.py:finish():358] shutting down directory watcher 2023-11-18 11:52:18,604 INFO SenderThread:16687 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_115015-iyvqyo9v/files/output.log 2023-11-18 11:52:18,606 INFO SenderThread:16687 [dir_watcher.py:finish():388] scan: /mnt/e/wandb/run-20231118_115015-iyvqyo9v/files 2023-11-18 11:52:18,608 INFO SenderThread:16687 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_115015-iyvqyo9v/files/conda-environment.yaml conda-environment.yaml 2023-11-18 11:52:18,610 INFO SenderThread:16687 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_115015-iyvqyo9v/files/config.yaml config.yaml 2023-11-18 11:52:18,615 INFO SenderThread:16687 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_115015-iyvqyo9v/files/output.log output.log 2023-11-18 11:52:18,619 INFO SenderThread:16687 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_115015-iyvqyo9v/files/requirements.txt requirements.txt 2023-11-18 11:52:18,624 INFO SenderThread:16687 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_115015-iyvqyo9v/files/wandb-metadata.json wandb-metadata.json 2023-11-18 11:52:18,625 INFO SenderThread:16687 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_115015-iyvqyo9v/files/wandb-summary.json wandb-summary.json 2023-11-18 11:52:18,629 INFO SenderThread:16687 [sender.py:transition_state():613] send defer: 10 2023-11-18 11:52:18,629 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:52:18,630 INFO HandlerThread:16687 [handler.py:handle_request_defer():172] handle defer: 10 2023-11-18 11:52:18,631 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: defer 2023-11-18 11:52:18,632 INFO SenderThread:16687 [sender.py:send_request_defer():609] handle sender defer: 10 2023-11-18 11:52:18,632 INFO SenderThread:16687 [file_pusher.py:finish():175] shutting down file pusher 2023-11-18 11:52:19,029 INFO wandb-upload_1:16687 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_115015-iyvqyo9v/files/config.yaml 2023-11-18 11:52:19,068 INFO wandb-upload_0:16687 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_115015-iyvqyo9v/files/conda-environment.yaml 2023-11-18 11:52:19,097 INFO wandb-upload_4:16687 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_115015-iyvqyo9v/files/wandb-summary.json 2023-11-18 11:52:19,129 INFO wandb-upload_2:16687 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_115015-iyvqyo9v/files/output.log 2023-11-18 11:52:19,133 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: poll_exit 2023-11-18 11:52:19,134 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: poll_exit 2023-11-18 11:52:19,158 INFO wandb-upload_3:16687 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_115015-iyvqyo9v/files/requirements.txt 2023-11-18 11:52:19,359 INFO Thread-11 (_thread_body):16687 [sender.py:transition_state():613] send defer: 11 2023-11-18 11:52:19,359 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:52:19,360 INFO HandlerThread:16687 [handler.py:handle_request_defer():172] handle defer: 11 2023-11-18 11:52:19,360 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: defer 2023-11-18 11:52:19,361 INFO SenderThread:16687 [sender.py:send_request_defer():609] handle sender defer: 11 2023-11-18 11:52:19,361 INFO SenderThread:16687 [file_pusher.py:join():181] waiting for file pusher 2023-11-18 11:52:19,361 INFO SenderThread:16687 [sender.py:transition_state():613] send defer: 12 2023-11-18 11:52:19,362 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:52:19,362 INFO HandlerThread:16687 [handler.py:handle_request_defer():172] handle defer: 12 2023-11-18 11:52:19,362 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: defer 2023-11-18 11:52:19,363 INFO SenderThread:16687 [sender.py:send_request_defer():609] handle sender defer: 12 2023-11-18 11:52:19,363 INFO SenderThread:16687 [file_stream.py:finish():595] file stream finish called 2023-11-18 11:52:19,862 INFO SenderThread:16687 [file_stream.py:finish():599] file stream finish is done 2023-11-18 11:52:19,863 INFO SenderThread:16687 [sender.py:transition_state():613] send defer: 13 2023-11-18 11:52:19,863 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:52:19,863 INFO HandlerThread:16687 [handler.py:handle_request_defer():172] handle defer: 13 2023-11-18 11:52:19,863 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: defer 2023-11-18 11:52:19,863 INFO SenderThread:16687 [sender.py:send_request_defer():609] handle sender defer: 13 2023-11-18 11:52:19,864 INFO SenderThread:16687 [sender.py:transition_state():613] send defer: 14 2023-11-18 11:52:19,864 DEBUG SenderThread:16687 [sender.py:send():380] send: final 2023-11-18 11:52:19,864 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:52:19,864 DEBUG SenderThread:16687 [sender.py:send():380] send: footer 2023-11-18 11:52:19,864 INFO HandlerThread:16687 [handler.py:handle_request_defer():172] handle defer: 14 2023-11-18 11:52:19,865 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: defer 2023-11-18 11:52:19,865 INFO SenderThread:16687 [sender.py:send_request_defer():609] handle sender defer: 14 2023-11-18 11:52:19,865 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: poll_exit 2023-11-18 11:52:19,866 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: poll_exit 2023-11-18 11:52:19,866 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: server_info 2023-11-18 11:52:19,866 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: get_summary 2023-11-18 11:52:19,867 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: poll_exit 2023-11-18 11:52:19,867 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: sampled_history 2023-11-18 11:52:19,867 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: poll_exit 2023-11-18 11:52:19,867 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 11:52:19,868 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: server_info 2023-11-18 11:52:19,868 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: job_info 2023-11-18 11:52:20,024 DEBUG SenderThread:16687 [sender.py:send_request():407] send_request: job_info 2023-11-18 11:52:20,025 INFO MainThread:16687 [wandb_run.py:_footer_history_summary_info():3760] rendering history 2023-11-18 11:52:20,025 INFO MainThread:16687 [wandb_run.py:_footer_history_summary_info():3792] rendering summary 2023-11-18 11:52:20,025 INFO MainThread:16687 [wandb_run.py:_footer_sync_info():3719] logging synced files 2023-11-18 11:52:20,026 DEBUG HandlerThread:16687 [handler.py:handle_request():146] handle_request: shutdown 2023-11-18 11:52:20,026 INFO HandlerThread:16687 [handler.py:finish():866] shutting down handler 2023-11-18 11:52:20,870 INFO WriterThread:16687 [datastore.py:close():294] close: /mnt/e/wandb/run-20231118_115015-iyvqyo9v/run-iyvqyo9v.wandb 2023-11-18 11:52:21,025 INFO SenderThread:16687 [sender.py:finish():1536] shutting down sender 2023-11-18 11:52:21,025 INFO SenderThread:16687 [file_pusher.py:finish():175] shutting down file pusher 2023-11-18 11:52:21,025 INFO SenderThread:16687 [file_pusher.py:join():181] waiting for file pusher