2023-11-18 11:54:22,182 INFO StreamThr :18234 [internal.py:wandb_internal():86] W&B internal server running at pid: 18234, started at: 2023-11-18 11:54:22.181139 2023-11-18 11:54:22,183 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: status 2023-11-18 11:54:22,186 INFO WriterThread:18234 [datastore.py:open_for_write():85] open: /mnt/e/wandb/run-20231118_115422-vkuemujk/run-vkuemujk.wandb 2023-11-18 11:54:22,188 DEBUG SenderThread:18234 [sender.py:send():380] send: header 2023-11-18 11:54:22,188 DEBUG SenderThread:18234 [sender.py:send():380] send: run 2023-11-18 11:54:22,588 INFO SenderThread:18234 [dir_watcher.py:__init__():211] watching files in: /mnt/e/wandb/run-20231118_115422-vkuemujk/files 2023-11-18 11:54:22,598 INFO SenderThread:18234 [sender.py:_start_run_threads():1124] run started: vkuemujk with start time 1700301262.181547 2023-11-18 11:54:22,599 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: check_version 2023-11-18 11:54:22,600 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: check_version 2023-11-18 11:54:22,645 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: run_start 2023-11-18 11:54:22,651 DEBUG HandlerThread:18234 [system_info.py:__init__():32] System info init 2023-11-18 11:54:22,651 DEBUG HandlerThread:18234 [system_info.py:__init__():47] System info init done 2023-11-18 11:54:22,651 INFO HandlerThread:18234 [system_monitor.py:start():194] Starting system monitor 2023-11-18 11:54:22,652 INFO SystemMonitor:18234 [system_monitor.py:_start():158] Starting system asset monitoring threads 2023-11-18 11:54:22,652 INFO HandlerThread:18234 [system_monitor.py:probe():214] Collecting system info 2023-11-18 11:54:22,652 INFO SystemMonitor:18234 [interfaces.py:start():190] Started cpu monitoring 2023-11-18 11:54:22,654 INFO SystemMonitor:18234 [interfaces.py:start():190] Started disk monitoring 2023-11-18 11:54:22,655 INFO SystemMonitor:18234 [interfaces.py:start():190] Started gpu monitoring 2023-11-18 11:54:22,656 INFO SystemMonitor:18234 [interfaces.py:start():190] Started memory monitoring 2023-11-18 11:54:22,656 INFO SystemMonitor:18234 [interfaces.py:start():190] Started network monitoring 2023-11-18 11:54:22,666 DEBUG HandlerThread:18234 [system_info.py:probe():196] Probing system 2023-11-18 11:54:22,760 DEBUG HandlerThread:18234 [gitlib.py:_init_repo():56] git repository is invalid 2023-11-18 11:54:22,761 DEBUG HandlerThread:18234 [system_info.py:probe():244] Probing system done 2023-11-18 11:54:22,761 DEBUG HandlerThread:18234 [system_monitor.py:probe():223] {'os': 'Linux-5.15.123.1-microsoft-standard-WSL2-x86_64-with-glibc2.35', 'python': '3.10.13', 'heartbeatAt': '2023-11-18T09:54:22.666667', 'startedAt': '2023-11-18T09:54:22.145260', 'docker': None, 'cuda': None, 'args': ('--deepspeed=ds_config.json', '--model_name_or_path=openai/whisper-large-v2', '--train_dataset_names', 'mozilla-foundation/common_voice_11_0', 'mozilla-foundation/common_voice_11_0', 'google/fleurs', 'Finnish-NLP/css10_audio_processed', 'Finnish-NLP/youtube_fi_110k_preprocessed', 'Finnish-NLP/voxpopuli_fi_audio_processed', 'Finnish-NLP/parliament_audio_processed', 'Finnish-NLP/aalto_eduskunta_asr_audio_processed', 'Finnish-NLP/fbc_dialog_audio_processed', 'Finnish-NLP/fbc_monolog_audio_processed', '--train_dataset_config_names', 'fi', 'fi', 'fi_fi', '', '', '', '', '', '', '', '--train_dataset_split_name', 'train', 'validation', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', '--train_dataset_text_column_names', 'sentence', 'sentence', 'raw_transcription', 'sentence', 'texts', 'sentence', 'sentence', 'sentence', 'sentence', 'sentence', '--eval_dataset_name=mozilla-foundation/common_voice_11_0', '--eval_dataset_config_name=fi', '--eval_split_name=test', '--language=finnish', '--model_index_name=Whisper Large V3 Finnish', '--max_steps=30000', '--output_dir=./', '--per_device_train_batch_size=1', '--per_device_eval_batch_size=2', '--gradient_accumulation_steps=2', '--logging_steps=25', '--learning_rate=3e-6', '--warmup_steps=1000', '--evaluation_strategy=steps', '--eval_steps=1000', '--save_strategy=steps', '--save_steps=1000', '--generation_max_length=225', '--length_column_name=input_length', '--max_duration_in_seconds=30', '--text_column_name=sentence', '--freeze_feature_encoder=False', '--report_to=wandb', '--metric_for_best_model=wer', '--greater_is_better=False', '--load_best_model_at_end', '--gradient_checkpointing', '--fp16', '--overwrite_output_dir', '--do_train', '--do_eval', '--predict_with_generate', '--do_normalize_eval', '--use_auth_token', '--push_to_hub'), 'state': 'running', 'program': '/mnt/e/run_speech_recognition_seq2seq_streaming.py', 'codePathLocal': 'run_speech_recognition_seq2seq_streaming.py', 'codePath': 'run_speech_recognition_seq2seq_streaming.py', 'host': 'DESKTOP-59O9VN1', 'username': 'rasmus', 'executable': '/home/rasmus/miniconda3/envs/WhisperFinetuneEnv/bin/python', 'cpu_count': 4, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3911.9949999999994, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 1006.853931427002, 'used': 37.274559020996094}}, 'gpu': 'NVIDIA GeForce RTX 4080', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 4080', 'memory_total': 17171480576}], 'memory': {'total': 54.92573928833008}} 2023-11-18 11:54:22,761 INFO HandlerThread:18234 [system_monitor.py:probe():224] Finished collecting system info 2023-11-18 11:54:22,761 INFO HandlerThread:18234 [system_monitor.py:probe():227] Publishing system info 2023-11-18 11:54:22,761 DEBUG HandlerThread:18234 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment 2023-11-18 11:54:22,763 DEBUG HandlerThread:18234 [system_info.py:_save_pip():68] Saving pip packages done 2023-11-18 11:54:22,763 DEBUG HandlerThread:18234 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment 2023-11-18 11:54:23,592 INFO Thread-12 :18234 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_115422-vkuemujk/files/conda-environment.yaml 2023-11-18 11:54:23,592 INFO Thread-12 :18234 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_115422-vkuemujk/files/requirements.txt 2023-11-18 11:54:24,735 DEBUG HandlerThread:18234 [system_info.py:_save_conda():87] Saving conda packages done 2023-11-18 11:54:24,737 INFO HandlerThread:18234 [system_monitor.py:probe():229] Finished publishing system info 2023-11-18 11:54:24,739 DEBUG SenderThread:18234 [sender.py:send():380] send: files 2023-11-18 11:54:24,739 INFO SenderThread:18234 [sender.py:_save_file():1380] saving file wandb-metadata.json with policy now 2023-11-18 11:54:24,746 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 11:54:24,747 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 11:54:24,747 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: stop_status 2023-11-18 11:54:24,968 DEBUG SenderThread:18234 [sender.py:send():380] send: telemetry 2023-11-18 11:54:24,969 DEBUG SenderThread:18234 [sender.py:send():380] send: config 2023-11-18 11:54:24,969 DEBUG SenderThread:18234 [sender.py:send():380] send: metric 2023-11-18 11:54:24,969 DEBUG SenderThread:18234 [sender.py:send():380] send: telemetry 2023-11-18 11:54:24,970 DEBUG SenderThread:18234 [sender.py:send():380] send: metric 2023-11-18 11:54:24,970 WARNING SenderThread:18234 [sender.py:send_metric():1331] Seen metric with glob (shouldn't happen) 2023-11-18 11:54:24,970 DEBUG SenderThread:18234 [sender.py:send():380] send: telemetry 2023-11-18 11:54:25,323 INFO wandb-upload_0:18234 [upload_job.py:push():131] Uploaded file /tmp/tmpxkho6qezwandb/gkbd0xom-wandb-metadata.json 2023-11-18 11:54:25,598 INFO Thread-12 :18234 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_115422-vkuemujk/files/conda-environment.yaml 2023-11-18 11:54:25,599 INFO Thread-12 :18234 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_115422-vkuemujk/files/output.log 2023-11-18 11:54:25,600 INFO Thread-12 :18234 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_115422-vkuemujk/files/wandb-metadata.json 2023-11-18 11:54:27,503 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:54:27,606 INFO Thread-12 :18234 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_115422-vkuemujk/files/output.log 2023-11-18 11:54:29,614 INFO Thread-12 :18234 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_115422-vkuemujk/files/output.log 2023-11-18 11:54:32,504 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:54:37,505 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:54:39,746 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 11:54:39,746 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 11:54:39,746 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: stop_status 2023-11-18 11:54:42,988 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:54:47,989 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:54:52,992 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:54:53,721 INFO Thread-12 :18234 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_115422-vkuemujk/files/config.yaml 2023-11-18 11:54:54,746 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 11:54:54,746 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: stop_status 2023-11-18 11:54:54,795 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 11:54:58,903 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:55:03,904 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:55:08,905 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:55:09,746 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 11:55:09,746 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: stop_status 2023-11-18 11:55:09,795 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 11:55:14,004 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:55:19,004 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:55:22,657 DEBUG SystemMonitor:18234 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2023-11-18 11:55:22,658 DEBUG SenderThread:18234 [sender.py:send():380] send: stats 2023-11-18 11:55:24,659 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:55:24,746 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 11:55:24,747 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: stop_status 2023-11-18 11:55:24,795 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 11:55:29,912 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:55:34,913 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:55:39,746 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 11:55:39,747 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: stop_status 2023-11-18 11:55:39,795 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 11:55:39,981 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:55:44,982 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:55:49,982 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:55:52,658 DEBUG SenderThread:18234 [sender.py:send():380] send: stats 2023-11-18 11:55:54,746 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 11:55:54,747 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: stop_status 2023-11-18 11:55:54,805 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 11:55:54,985 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:55:59,986 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:56:04,986 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:56:09,746 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: stop_status 2023-11-18 11:56:09,747 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: stop_status 2023-11-18 11:56:09,795 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 11:56:10,931 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:56:15,932 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:56:20,069 INFO Thread-12 :18234 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_115422-vkuemujk/files/output.log 2023-11-18 11:56:21,073 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:56:21,077 DEBUG SenderThread:18234 [sender.py:send():380] send: exit 2023-11-18 11:56:21,078 INFO SenderThread:18234 [sender.py:send_exit():585] handling exit code: 1 2023-11-18 11:56:21,078 INFO SenderThread:18234 [sender.py:send_exit():587] handling runtime: 118 2023-11-18 11:56:21,089 INFO SenderThread:18234 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 11:56:21,090 INFO SenderThread:18234 [sender.py:send_exit():593] send defer 2023-11-18 11:56:21,090 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:56:21,091 INFO HandlerThread:18234 [handler.py:handle_request_defer():172] handle defer: 0 2023-11-18 11:56:21,092 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: defer 2023-11-18 11:56:21,092 INFO SenderThread:18234 [sender.py:send_request_defer():609] handle sender defer: 0 2023-11-18 11:56:21,092 INFO SenderThread:18234 [sender.py:transition_state():613] send defer: 1 2023-11-18 11:56:21,093 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:56:21,093 INFO HandlerThread:18234 [handler.py:handle_request_defer():172] handle defer: 1 2023-11-18 11:56:21,093 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: defer 2023-11-18 11:56:21,093 INFO SenderThread:18234 [sender.py:send_request_defer():609] handle sender defer: 1 2023-11-18 11:56:21,094 INFO SenderThread:18234 [sender.py:transition_state():613] send defer: 2 2023-11-18 11:56:21,094 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:56:21,094 INFO HandlerThread:18234 [handler.py:handle_request_defer():172] handle defer: 2 2023-11-18 11:56:21,094 INFO HandlerThread:18234 [system_monitor.py:finish():203] Stopping system monitor 2023-11-18 11:56:21,095 DEBUG SystemMonitor:18234 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2023-11-18 11:56:21,095 DEBUG SystemMonitor:18234 [system_monitor.py:_start():183] Publishing last batch of metrics 2023-11-18 11:56:21,095 INFO HandlerThread:18234 [interfaces.py:finish():202] Joined cpu monitor 2023-11-18 11:56:21,096 INFO HandlerThread:18234 [interfaces.py:finish():202] Joined disk monitor 2023-11-18 11:56:21,104 INFO HandlerThread:18234 [interfaces.py:finish():202] Joined gpu monitor 2023-11-18 11:56:21,104 INFO HandlerThread:18234 [interfaces.py:finish():202] Joined memory monitor 2023-11-18 11:56:21,104 INFO HandlerThread:18234 [interfaces.py:finish():202] Joined network monitor 2023-11-18 11:56:21,105 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: defer 2023-11-18 11:56:21,110 INFO SenderThread:18234 [sender.py:send_request_defer():609] handle sender defer: 2 2023-11-18 11:56:21,114 INFO SenderThread:18234 [sender.py:transition_state():613] send defer: 3 2023-11-18 11:56:21,115 DEBUG SenderThread:18234 [sender.py:send():380] send: stats 2023-11-18 11:56:21,115 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:56:21,115 INFO HandlerThread:18234 [handler.py:handle_request_defer():172] handle defer: 3 2023-11-18 11:56:21,116 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: defer 2023-11-18 11:56:21,116 INFO SenderThread:18234 [sender.py:send_request_defer():609] handle sender defer: 3 2023-11-18 11:56:21,116 INFO SenderThread:18234 [sender.py:transition_state():613] send defer: 4 2023-11-18 11:56:21,117 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:56:21,117 INFO HandlerThread:18234 [handler.py:handle_request_defer():172] handle defer: 4 2023-11-18 11:56:21,117 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: defer 2023-11-18 11:56:21,118 INFO SenderThread:18234 [sender.py:send_request_defer():609] handle sender defer: 4 2023-11-18 11:56:21,118 INFO SenderThread:18234 [sender.py:transition_state():613] send defer: 5 2023-11-18 11:56:21,118 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:56:21,118 INFO HandlerThread:18234 [handler.py:handle_request_defer():172] handle defer: 5 2023-11-18 11:56:21,119 DEBUG SenderThread:18234 [sender.py:send():380] send: summary 2023-11-18 11:56:21,120 INFO SenderThread:18234 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end 2023-11-18 11:56:21,121 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: defer 2023-11-18 11:56:21,121 INFO SenderThread:18234 [sender.py:send_request_defer():609] handle sender defer: 5 2023-11-18 11:56:21,121 INFO SenderThread:18234 [sender.py:transition_state():613] send defer: 6 2023-11-18 11:56:21,121 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:56:21,122 INFO HandlerThread:18234 [handler.py:handle_request_defer():172] handle defer: 6 2023-11-18 11:56:21,122 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: defer 2023-11-18 11:56:21,122 INFO SenderThread:18234 [sender.py:send_request_defer():609] handle sender defer: 6 2023-11-18 11:56:21,122 INFO SenderThread:18234 [sender.py:transition_state():613] send defer: 7 2023-11-18 11:56:21,123 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: status_report 2023-11-18 11:56:21,123 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:56:21,123 INFO HandlerThread:18234 [handler.py:handle_request_defer():172] handle defer: 7 2023-11-18 11:56:21,123 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: defer 2023-11-18 11:56:21,124 INFO SenderThread:18234 [sender.py:send_request_defer():609] handle sender defer: 7 2023-11-18 11:56:22,077 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: poll_exit 2023-11-18 11:56:22,078 INFO Thread-12 :18234 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_115422-vkuemujk/files/output.log 2023-11-18 11:56:22,079 INFO Thread-12 :18234 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_115422-vkuemujk/files/wandb-summary.json 2023-11-18 11:56:23,110 INFO SenderThread:18234 [sender.py:transition_state():613] send defer: 8 2023-11-18 11:56:23,110 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: poll_exit 2023-11-18 11:56:23,110 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:56:23,111 INFO HandlerThread:18234 [handler.py:handle_request_defer():172] handle defer: 8 2023-11-18 11:56:23,111 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: defer 2023-11-18 11:56:23,111 INFO SenderThread:18234 [sender.py:send_request_defer():609] handle sender defer: 8 2023-11-18 11:56:23,112 INFO SenderThread:18234 [job_builder.py:build():298] Attempting to build job artifact 2023-11-18 11:56:23,116 INFO SenderThread:18234 [job_builder.py:_get_source_type():439] no source found 2023-11-18 11:56:23,116 INFO SenderThread:18234 [sender.py:transition_state():613] send defer: 9 2023-11-18 11:56:23,117 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:56:23,117 INFO HandlerThread:18234 [handler.py:handle_request_defer():172] handle defer: 9 2023-11-18 11:56:23,117 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: defer 2023-11-18 11:56:23,117 INFO SenderThread:18234 [sender.py:send_request_defer():609] handle sender defer: 9 2023-11-18 11:56:23,117 INFO SenderThread:18234 [dir_watcher.py:finish():358] shutting down directory watcher 2023-11-18 11:56:24,087 INFO SenderThread:18234 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_115422-vkuemujk/files/output.log 2023-11-18 11:56:24,094 INFO SenderThread:18234 [dir_watcher.py:finish():388] scan: /mnt/e/wandb/run-20231118_115422-vkuemujk/files 2023-11-18 11:56:24,096 INFO SenderThread:18234 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_115422-vkuemujk/files/conda-environment.yaml conda-environment.yaml 2023-11-18 11:56:24,099 INFO SenderThread:18234 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_115422-vkuemujk/files/config.yaml config.yaml 2023-11-18 11:56:24,103 INFO SenderThread:18234 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_115422-vkuemujk/files/output.log output.log 2023-11-18 11:56:24,107 INFO SenderThread:18234 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_115422-vkuemujk/files/requirements.txt requirements.txt 2023-11-18 11:56:24,112 INFO SenderThread:18234 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_115422-vkuemujk/files/wandb-metadata.json wandb-metadata.json 2023-11-18 11:56:24,113 INFO SenderThread:18234 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_115422-vkuemujk/files/wandb-summary.json wandb-summary.json 2023-11-18 11:56:24,117 INFO SenderThread:18234 [sender.py:transition_state():613] send defer: 10 2023-11-18 11:56:24,118 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:56:24,118 INFO HandlerThread:18234 [handler.py:handle_request_defer():172] handle defer: 10 2023-11-18 11:56:24,118 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: defer 2023-11-18 11:56:24,119 INFO SenderThread:18234 [sender.py:send_request_defer():609] handle sender defer: 10 2023-11-18 11:56:24,120 INFO SenderThread:18234 [file_pusher.py:finish():175] shutting down file pusher 2023-11-18 11:56:24,511 INFO wandb-upload_1:18234 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_115422-vkuemujk/files/config.yaml 2023-11-18 11:56:24,536 INFO wandb-upload_0:18234 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_115422-vkuemujk/files/conda-environment.yaml 2023-11-18 11:56:24,582 INFO wandb-upload_4:18234 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_115422-vkuemujk/files/wandb-summary.json 2023-11-18 11:56:24,599 INFO wandb-upload_2:18234 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_115422-vkuemujk/files/output.log 2023-11-18 11:56:24,603 INFO wandb-upload_3:18234 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_115422-vkuemujk/files/requirements.txt 2023-11-18 11:56:24,803 INFO Thread-11 (_thread_body):18234 [sender.py:transition_state():613] send defer: 11 2023-11-18 11:56:24,804 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:56:24,804 INFO HandlerThread:18234 [handler.py:handle_request_defer():172] handle defer: 11 2023-11-18 11:56:24,804 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: defer 2023-11-18 11:56:24,805 INFO SenderThread:18234 [sender.py:send_request_defer():609] handle sender defer: 11 2023-11-18 11:56:24,805 INFO SenderThread:18234 [file_pusher.py:join():181] waiting for file pusher 2023-11-18 11:56:24,805 INFO SenderThread:18234 [sender.py:transition_state():613] send defer: 12 2023-11-18 11:56:24,805 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:56:24,805 INFO HandlerThread:18234 [handler.py:handle_request_defer():172] handle defer: 12 2023-11-18 11:56:24,806 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: defer 2023-11-18 11:56:24,806 INFO SenderThread:18234 [sender.py:send_request_defer():609] handle sender defer: 12 2023-11-18 11:56:24,806 INFO SenderThread:18234 [file_stream.py:finish():595] file stream finish called 2023-11-18 11:56:25,351 INFO SenderThread:18234 [file_stream.py:finish():599] file stream finish is done 2023-11-18 11:56:25,351 INFO SenderThread:18234 [sender.py:transition_state():613] send defer: 13 2023-11-18 11:56:25,351 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:56:25,352 INFO HandlerThread:18234 [handler.py:handle_request_defer():172] handle defer: 13 2023-11-18 11:56:25,352 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: defer 2023-11-18 11:56:25,352 INFO SenderThread:18234 [sender.py:send_request_defer():609] handle sender defer: 13 2023-11-18 11:56:25,352 INFO SenderThread:18234 [sender.py:transition_state():613] send defer: 14 2023-11-18 11:56:25,353 DEBUG SenderThread:18234 [sender.py:send():380] send: final 2023-11-18 11:56:25,353 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: defer 2023-11-18 11:56:25,353 DEBUG SenderThread:18234 [sender.py:send():380] send: footer 2023-11-18 11:56:25,353 INFO HandlerThread:18234 [handler.py:handle_request_defer():172] handle defer: 14 2023-11-18 11:56:25,354 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: defer 2023-11-18 11:56:25,354 INFO SenderThread:18234 [sender.py:send_request_defer():609] handle sender defer: 14 2023-11-18 11:56:25,354 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: poll_exit 2023-11-18 11:56:25,355 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: server_info 2023-11-18 11:56:25,355 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: poll_exit 2023-11-18 11:56:25,355 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: get_summary 2023-11-18 11:56:25,355 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: server_info 2023-11-18 11:56:25,355 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: sampled_history 2023-11-18 11:56:25,357 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: internal_messages 2023-11-18 11:56:25,357 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: job_info 2023-11-18 11:56:25,503 DEBUG SenderThread:18234 [sender.py:send_request():407] send_request: job_info 2023-11-18 11:56:25,504 INFO MainThread:18234 [wandb_run.py:_footer_history_summary_info():3760] rendering history 2023-11-18 11:56:25,504 INFO MainThread:18234 [wandb_run.py:_footer_history_summary_info():3792] rendering summary 2023-11-18 11:56:25,504 INFO MainThread:18234 [wandb_run.py:_footer_sync_info():3719] logging synced files 2023-11-18 11:56:25,505 DEBUG HandlerThread:18234 [handler.py:handle_request():146] handle_request: shutdown 2023-11-18 11:56:25,505 INFO HandlerThread:18234 [handler.py:finish():866] shutting down handler 2023-11-18 11:56:26,358 INFO WriterThread:18234 [datastore.py:close():294] close: /mnt/e/wandb/run-20231118_115422-vkuemujk/run-vkuemujk.wandb 2023-11-18 11:56:26,504 INFO SenderThread:18234 [sender.py:finish():1536] shutting down sender 2023-11-18 11:56:26,504 INFO SenderThread:18234 [file_pusher.py:finish():175] shutting down file pusher 2023-11-18 11:56:26,504 INFO SenderThread:18234 [file_pusher.py:join():181] waiting for file pusher