RASMUS's picture
Training in progress, step 1000
878dbce
2023-11-18 11:37:38,681 INFO StreamThr :11968 [internal.py:wandb_internal():86] W&B internal server running at pid: 11968, started at: 2023-11-18 11:37:38.679482
2023-11-18 11:37:38,682 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status
2023-11-18 11:37:38,687 INFO WriterThread:11968 [datastore.py:open_for_write():85] open: /mnt/e/wandb/run-20231118_113738-lw67dmi9/run-lw67dmi9.wandb
2023-11-18 11:37:38,689 DEBUG SenderThread:11968 [sender.py:send():380] send: header
2023-11-18 11:37:38,689 DEBUG SenderThread:11968 [sender.py:send():380] send: run
2023-11-18 11:37:40,207 INFO SenderThread:11968 [dir_watcher.py:__init__():211] watching files in: /mnt/e/wandb/run-20231118_113738-lw67dmi9/files
2023-11-18 11:37:40,218 INFO SenderThread:11968 [sender.py:_start_run_threads():1124] run started: lw67dmi9 with start time 1700300258.68007
2023-11-18 11:37:40,222 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: check_version
2023-11-18 11:37:40,222 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: check_version
2023-11-18 11:37:40,284 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: run_start
2023-11-18 11:37:40,300 DEBUG HandlerThread:11968 [system_info.py:__init__():32] System info init
2023-11-18 11:37:40,301 DEBUG HandlerThread:11968 [system_info.py:__init__():47] System info init done
2023-11-18 11:37:40,301 INFO HandlerThread:11968 [system_monitor.py:start():194] Starting system monitor
2023-11-18 11:37:40,302 INFO SystemMonitor:11968 [system_monitor.py:_start():158] Starting system asset monitoring threads
2023-11-18 11:37:40,303 INFO HandlerThread:11968 [system_monitor.py:probe():214] Collecting system info
2023-11-18 11:37:40,304 INFO SystemMonitor:11968 [interfaces.py:start():190] Started cpu monitoring
2023-11-18 11:37:40,308 INFO SystemMonitor:11968 [interfaces.py:start():190] Started disk monitoring
2023-11-18 11:37:40,308 INFO SystemMonitor:11968 [interfaces.py:start():190] Started gpu monitoring
2023-11-18 11:37:40,310 INFO SystemMonitor:11968 [interfaces.py:start():190] Started memory monitoring
2023-11-18 11:37:40,311 INFO SystemMonitor:11968 [interfaces.py:start():190] Started network monitoring
2023-11-18 11:37:40,321 DEBUG HandlerThread:11968 [system_info.py:probe():196] Probing system
2023-11-18 11:37:40,510 DEBUG HandlerThread:11968 [gitlib.py:_init_repo():56] git repository is invalid
2023-11-18 11:37:40,511 DEBUG HandlerThread:11968 [system_info.py:probe():244] Probing system done
2023-11-18 11:37:40,511 DEBUG HandlerThread:11968 [system_monitor.py:probe():223] {'os': 'Linux-5.15.123.1-microsoft-standard-WSL2-x86_64-with-glibc2.35', 'python': '3.10.13', 'heartbeatAt': '2023-11-18T09:37:40.321959', 'startedAt': '2023-11-18T09:37:38.629248', 'docker': None, 'cuda': None, 'args': ('--deepspeed=ds_config.json', '--model_name_or_path=openai/whisper-large-v2', '--train_dataset_names', 'mozilla-foundation/common_voice_11_0', 'mozilla-foundation/common_voice_11_0', 'google/fleurs', 'Finnish-NLP/css10_audio_processed', 'Finnish-NLP/youtube_fi_110k_preprocessed', 'Finnish-NLP/voxpopuli_fi_audio_processed', 'Finnish-NLP/parliament_audio_processed', 'Finnish-NLP/aalto_eduskunta_asr_audio_processed', 'Finnish-NLP/fbc_dialog_audio_processed', 'Finnish-NLP/fbc_monolog_audio_processed', '--train_dataset_config_names', 'fi', 'fi', 'fi_fi', '', '', '', '', '', '', '', '--train_dataset_split_name', 'train', 'validation', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', '--train_dataset_text_column_names', 'sentence', 'sentence', 'raw_transcription', 'sentence', 'texts', 'sentence', 'sentence', 'sentence', 'sentence', 'sentence', '--eval_dataset_name=mozilla-foundation/common_voice_11_0', '--eval_dataset_config_name=fi', '--eval_split_name=test', '--language=finnish', '--model_index_name=Whisper Large V3 Finnish', '--max_steps=30000', '--output_dir=./', '--per_device_train_batch_size=16', '--per_device_eval_batch_size=8', '--gradient_accumulation_steps=4', '--logging_steps=25', '--learning_rate=3e-6', '--warmup_steps=1000', '--evaluation_strategy=steps', '--eval_steps=1000', '--save_strategy=steps', '--save_steps=1000', '--generation_max_length=225', '--length_column_name=input_length', '--max_duration_in_seconds=30', '--text_column_name=sentence', '--freeze_feature_encoder=False', '--report_to=wandb', '--metric_for_best_model=wer', '--greater_is_better=False', '--load_best_model_at_end', '--gradient_checkpointing', '--fp16', '--overwrite_output_dir', '--do_train', '--do_eval', '--predict_with_generate', '--do_normalize_eval', '--use_auth_token', '--push_to_hub'), 'state': 'running', 'program': '/mnt/e/run_speech_recognition_seq2seq_streaming.py', 'codePathLocal': 'run_speech_recognition_seq2seq_streaming.py', 'codePath': 'run_speech_recognition_seq2seq_streaming.py', 'host': 'DESKTOP-59O9VN1', 'username': 'rasmus', 'executable': '/home/rasmus/miniconda3/envs/WhisperFinetuneEnv/bin/python', 'cpu_count': 4, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3911.9949999999994, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 1006.853931427002, 'used': 37.273719787597656}}, 'gpu': 'NVIDIA GeForce RTX 4080', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 4080', 'memory_total': 17171480576}], 'memory': {'total': 54.92573928833008}}
2023-11-18 11:37:40,511 INFO HandlerThread:11968 [system_monitor.py:probe():224] Finished collecting system info
2023-11-18 11:37:40,511 INFO HandlerThread:11968 [system_monitor.py:probe():227] Publishing system info
2023-11-18 11:37:40,512 DEBUG HandlerThread:11968 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment
2023-11-18 11:37:40,513 DEBUG HandlerThread:11968 [system_info.py:_save_pip():68] Saving pip packages done
2023-11-18 11:37:40,513 DEBUG HandlerThread:11968 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment
2023-11-18 11:37:41,211 INFO Thread-12 :11968 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_113738-lw67dmi9/files/conda-environment.yaml
2023-11-18 11:37:41,212 INFO Thread-12 :11968 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_113738-lw67dmi9/files/requirements.txt
2023-11-18 11:37:42,658 DEBUG HandlerThread:11968 [system_info.py:_save_conda():87] Saving conda packages done
2023-11-18 11:37:42,663 INFO HandlerThread:11968 [system_monitor.py:probe():229] Finished publishing system info
2023-11-18 11:37:42,666 DEBUG SenderThread:11968 [sender.py:send():380] send: files
2023-11-18 11:37:42,667 INFO SenderThread:11968 [sender.py:_save_file():1380] saving file wandb-metadata.json with policy now
2023-11-18 11:37:42,674 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:37:42,675 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:37:42,678 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:37:42,842 DEBUG SenderThread:11968 [sender.py:send():380] send: telemetry
2023-11-18 11:37:42,843 DEBUG SenderThread:11968 [sender.py:send():380] send: config
2023-11-18 11:37:42,844 DEBUG SenderThread:11968 [sender.py:send():380] send: metric
2023-11-18 11:37:42,845 DEBUG SenderThread:11968 [sender.py:send():380] send: telemetry
2023-11-18 11:37:42,846 DEBUG SenderThread:11968 [sender.py:send():380] send: metric
2023-11-18 11:37:42,846 WARNING SenderThread:11968 [sender.py:send_metric():1331] Seen metric with glob (shouldn't happen)
2023-11-18 11:37:42,847 DEBUG SenderThread:11968 [sender.py:send():380] send: telemetry
2023-11-18 11:37:43,220 INFO Thread-12 :11968 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_113738-lw67dmi9/files/conda-environment.yaml
2023-11-18 11:37:43,221 INFO Thread-12 :11968 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_113738-lw67dmi9/files/output.log
2023-11-18 11:37:43,221 INFO Thread-12 :11968 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_113738-lw67dmi9/files/wandb-metadata.json
2023-11-18 11:37:43,395 INFO wandb-upload_0:11968 [upload_job.py:push():131] Uploaded file /tmp/tmp4wxhkzl_wandb/fujba36k-wandb-metadata.json
2023-11-18 11:37:44,538 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:37:45,233 INFO Thread-12 :11968 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_113738-lw67dmi9/files/output.log
2023-11-18 11:37:47,242 INFO Thread-12 :11968 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_113738-lw67dmi9/files/output.log
2023-11-18 11:37:49,257 INFO Thread-12 :11968 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_113738-lw67dmi9/files/output.log
2023-11-18 11:37:50,422 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:37:51,270 INFO Thread-12 :11968 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_113738-lw67dmi9/files/output.log
2023-11-18 11:37:55,423 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:37:57,674 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:37:57,675 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:37:57,675 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:38:00,851 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:38:05,851 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:38:10,855 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:38:11,384 INFO Thread-12 :11968 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_113738-lw67dmi9/files/config.yaml
2023-11-18 11:38:12,675 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:38:12,676 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:38:12,716 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:38:16,876 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:38:21,877 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:38:26,878 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:38:27,675 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:38:27,676 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:38:27,715 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:38:31,891 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:38:36,892 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:38:40,313 DEBUG SystemMonitor:11968 [system_monitor.py:_start():172] Starting system metrics aggregation loop
2023-11-18 11:38:40,315 DEBUG SenderThread:11968 [sender.py:send():380] send: stats
2023-11-18 11:38:42,316 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:38:42,675 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:38:42,676 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:38:42,726 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:38:47,909 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:38:52,910 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:38:57,675 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:38:57,676 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:38:57,715 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:38:58,841 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:39:03,842 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:39:08,843 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:39:10,316 DEBUG SenderThread:11968 [sender.py:send():380] send: stats
2023-11-18 11:39:12,675 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:39:12,676 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:39:12,725 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:39:13,881 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:39:18,882 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:39:23,883 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:39:27,676 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:39:27,677 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:39:27,677 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:39:28,910 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:39:33,911 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:39:38,912 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:39:40,316 DEBUG SenderThread:11968 [sender.py:send():380] send: stats
2023-11-18 11:39:42,676 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:39:42,677 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:39:42,726 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:39:44,884 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:39:49,885 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:39:54,886 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:39:57,676 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:39:57,676 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:39:57,725 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:40:00,862 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:40:05,863 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:40:10,316 DEBUG SenderThread:11968 [sender.py:send():380] send: stats
2023-11-18 11:40:11,316 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:40:12,676 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: stop_status
2023-11-18 11:40:12,677 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: stop_status
2023-11-18 11:40:12,725 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:40:16,849 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:40:22,344 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:40:23,148 INFO Thread-12 :11968 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_113738-lw67dmi9/files/output.log
2023-11-18 11:40:27,345 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:40:27,623 DEBUG SenderThread:11968 [sender.py:send():380] send: exit
2023-11-18 11:40:27,623 INFO SenderThread:11968 [sender.py:send_exit():585] handling exit code: 1
2023-11-18 11:40:27,623 INFO SenderThread:11968 [sender.py:send_exit():587] handling runtime: 167
2023-11-18 11:40:27,635 INFO SenderThread:11968 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end
2023-11-18 11:40:27,636 INFO SenderThread:11968 [sender.py:send_exit():593] send defer
2023-11-18 11:40:27,636 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:40:27,636 INFO HandlerThread:11968 [handler.py:handle_request_defer():172] handle defer: 0
2023-11-18 11:40:27,637 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: defer
2023-11-18 11:40:27,637 INFO SenderThread:11968 [sender.py:send_request_defer():609] handle sender defer: 0
2023-11-18 11:40:27,637 INFO SenderThread:11968 [sender.py:transition_state():613] send defer: 1
2023-11-18 11:40:27,638 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:40:27,638 INFO HandlerThread:11968 [handler.py:handle_request_defer():172] handle defer: 1
2023-11-18 11:40:27,638 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: defer
2023-11-18 11:40:27,639 INFO SenderThread:11968 [sender.py:send_request_defer():609] handle sender defer: 1
2023-11-18 11:40:27,639 INFO SenderThread:11968 [sender.py:transition_state():613] send defer: 2
2023-11-18 11:40:27,639 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:40:27,640 INFO HandlerThread:11968 [handler.py:handle_request_defer():172] handle defer: 2
2023-11-18 11:40:27,640 INFO HandlerThread:11968 [system_monitor.py:finish():203] Stopping system monitor
2023-11-18 11:40:27,640 DEBUG SystemMonitor:11968 [system_monitor.py:_start():179] Finished system metrics aggregation loop
2023-11-18 11:40:27,641 INFO HandlerThread:11968 [interfaces.py:finish():202] Joined cpu monitor
2023-11-18 11:40:27,641 DEBUG SystemMonitor:11968 [system_monitor.py:_start():183] Publishing last batch of metrics
2023-11-18 11:40:27,642 INFO HandlerThread:11968 [interfaces.py:finish():202] Joined disk monitor
2023-11-18 11:40:27,652 INFO HandlerThread:11968 [interfaces.py:finish():202] Joined gpu monitor
2023-11-18 11:40:27,652 INFO HandlerThread:11968 [interfaces.py:finish():202] Joined memory monitor
2023-11-18 11:40:27,652 INFO HandlerThread:11968 [interfaces.py:finish():202] Joined network monitor
2023-11-18 11:40:27,653 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: defer
2023-11-18 11:40:27,653 INFO SenderThread:11968 [sender.py:send_request_defer():609] handle sender defer: 2
2023-11-18 11:40:27,653 INFO SenderThread:11968 [sender.py:transition_state():613] send defer: 3
2023-11-18 11:40:27,654 DEBUG SenderThread:11968 [sender.py:send():380] send: stats
2023-11-18 11:40:27,654 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:40:27,654 INFO HandlerThread:11968 [handler.py:handle_request_defer():172] handle defer: 3
2023-11-18 11:40:27,655 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: defer
2023-11-18 11:40:27,655 INFO SenderThread:11968 [sender.py:send_request_defer():609] handle sender defer: 3
2023-11-18 11:40:27,655 INFO SenderThread:11968 [sender.py:transition_state():613] send defer: 4
2023-11-18 11:40:27,655 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:40:27,656 INFO HandlerThread:11968 [handler.py:handle_request_defer():172] handle defer: 4
2023-11-18 11:40:27,656 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: defer
2023-11-18 11:40:27,656 INFO SenderThread:11968 [sender.py:send_request_defer():609] handle sender defer: 4
2023-11-18 11:40:27,656 INFO SenderThread:11968 [sender.py:transition_state():613] send defer: 5
2023-11-18 11:40:27,657 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:40:27,657 INFO HandlerThread:11968 [handler.py:handle_request_defer():172] handle defer: 5
2023-11-18 11:40:27,657 DEBUG SenderThread:11968 [sender.py:send():380] send: summary
2023-11-18 11:40:27,659 INFO SenderThread:11968 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end
2023-11-18 11:40:27,660 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: defer
2023-11-18 11:40:27,660 INFO SenderThread:11968 [sender.py:send_request_defer():609] handle sender defer: 5
2023-11-18 11:40:27,661 INFO SenderThread:11968 [sender.py:transition_state():613] send defer: 6
2023-11-18 11:40:27,661 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:40:27,661 INFO HandlerThread:11968 [handler.py:handle_request_defer():172] handle defer: 6
2023-11-18 11:40:27,661 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: defer
2023-11-18 11:40:27,662 INFO SenderThread:11968 [sender.py:send_request_defer():609] handle sender defer: 6
2023-11-18 11:40:27,662 INFO SenderThread:11968 [sender.py:transition_state():613] send defer: 7
2023-11-18 11:40:27,662 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: status_report
2023-11-18 11:40:27,662 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:40:27,663 INFO HandlerThread:11968 [handler.py:handle_request_defer():172] handle defer: 7
2023-11-18 11:40:27,663 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: defer
2023-11-18 11:40:27,663 INFO SenderThread:11968 [sender.py:send_request_defer():609] handle sender defer: 7
2023-11-18 11:40:28,177 INFO Thread-12 :11968 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_113738-lw67dmi9/files/wandb-summary.json
2023-11-18 11:40:28,622 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: poll_exit
2023-11-18 11:40:29,049 INFO SenderThread:11968 [sender.py:transition_state():613] send defer: 8
2023-11-18 11:40:29,050 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: poll_exit
2023-11-18 11:40:29,050 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:40:29,051 INFO HandlerThread:11968 [handler.py:handle_request_defer():172] handle defer: 8
2023-11-18 11:40:29,052 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: defer
2023-11-18 11:40:29,054 INFO SenderThread:11968 [sender.py:send_request_defer():609] handle sender defer: 8
2023-11-18 11:40:29,054 INFO SenderThread:11968 [job_builder.py:build():298] Attempting to build job artifact
2023-11-18 11:40:29,063 INFO SenderThread:11968 [job_builder.py:_get_source_type():439] no source found
2023-11-18 11:40:29,063 INFO SenderThread:11968 [sender.py:transition_state():613] send defer: 9
2023-11-18 11:40:29,064 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:40:29,064 INFO HandlerThread:11968 [handler.py:handle_request_defer():172] handle defer: 9
2023-11-18 11:40:29,064 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: defer
2023-11-18 11:40:29,065 INFO SenderThread:11968 [sender.py:send_request_defer():609] handle sender defer: 9
2023-11-18 11:40:29,065 INFO SenderThread:11968 [dir_watcher.py:finish():358] shutting down directory watcher
2023-11-18 11:40:29,191 INFO SenderThread:11968 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_113738-lw67dmi9/files/output.log
2023-11-18 11:40:29,192 INFO SenderThread:11968 [dir_watcher.py:finish():388] scan: /mnt/e/wandb/run-20231118_113738-lw67dmi9/files
2023-11-18 11:40:29,194 INFO SenderThread:11968 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_113738-lw67dmi9/files/conda-environment.yaml conda-environment.yaml
2023-11-18 11:40:29,197 INFO SenderThread:11968 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_113738-lw67dmi9/files/config.yaml config.yaml
2023-11-18 11:40:29,202 INFO SenderThread:11968 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_113738-lw67dmi9/files/output.log output.log
2023-11-18 11:40:29,207 INFO SenderThread:11968 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_113738-lw67dmi9/files/requirements.txt requirements.txt
2023-11-18 11:40:29,214 INFO SenderThread:11968 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_113738-lw67dmi9/files/wandb-metadata.json wandb-metadata.json
2023-11-18 11:40:29,214 INFO SenderThread:11968 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_113738-lw67dmi9/files/wandb-summary.json wandb-summary.json
2023-11-18 11:40:29,219 INFO SenderThread:11968 [sender.py:transition_state():613] send defer: 10
2023-11-18 11:40:29,220 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:40:29,220 INFO HandlerThread:11968 [handler.py:handle_request_defer():172] handle defer: 10
2023-11-18 11:40:29,221 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: defer
2023-11-18 11:40:29,221 INFO SenderThread:11968 [sender.py:send_request_defer():609] handle sender defer: 10
2023-11-18 11:40:29,222 INFO SenderThread:11968 [file_pusher.py:finish():175] shutting down file pusher
2023-11-18 11:40:29,588 INFO wandb-upload_0:11968 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_113738-lw67dmi9/files/conda-environment.yaml
2023-11-18 11:40:29,623 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: poll_exit
2023-11-18 11:40:29,623 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: poll_exit
2023-11-18 11:40:29,739 INFO wandb-upload_4:11968 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_113738-lw67dmi9/files/wandb-summary.json
2023-11-18 11:40:29,745 INFO wandb-upload_1:11968 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_113738-lw67dmi9/files/config.yaml
2023-11-18 11:40:29,754 INFO wandb-upload_3:11968 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_113738-lw67dmi9/files/requirements.txt
2023-11-18 11:40:29,848 INFO wandb-upload_2:11968 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_113738-lw67dmi9/files/output.log
2023-11-18 11:40:30,049 INFO Thread-11 (_thread_body):11968 [sender.py:transition_state():613] send defer: 11
2023-11-18 11:40:30,050 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:40:30,051 INFO HandlerThread:11968 [handler.py:handle_request_defer():172] handle defer: 11
2023-11-18 11:40:30,052 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: defer
2023-11-18 11:40:30,053 INFO SenderThread:11968 [sender.py:send_request_defer():609] handle sender defer: 11
2023-11-18 11:40:30,053 INFO SenderThread:11968 [file_pusher.py:join():181] waiting for file pusher
2023-11-18 11:40:30,054 INFO SenderThread:11968 [sender.py:transition_state():613] send defer: 12
2023-11-18 11:40:30,054 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:40:30,055 INFO HandlerThread:11968 [handler.py:handle_request_defer():172] handle defer: 12
2023-11-18 11:40:30,055 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: defer
2023-11-18 11:40:30,056 INFO SenderThread:11968 [sender.py:send_request_defer():609] handle sender defer: 12
2023-11-18 11:40:30,056 INFO SenderThread:11968 [file_stream.py:finish():595] file stream finish called
2023-11-18 11:40:30,588 INFO SenderThread:11968 [file_stream.py:finish():599] file stream finish is done
2023-11-18 11:40:30,589 INFO SenderThread:11968 [sender.py:transition_state():613] send defer: 13
2023-11-18 11:40:30,589 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:40:30,589 INFO HandlerThread:11968 [handler.py:handle_request_defer():172] handle defer: 13
2023-11-18 11:40:30,590 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: defer
2023-11-18 11:40:30,590 INFO SenderThread:11968 [sender.py:send_request_defer():609] handle sender defer: 13
2023-11-18 11:40:30,590 INFO SenderThread:11968 [sender.py:transition_state():613] send defer: 14
2023-11-18 11:40:30,590 DEBUG SenderThread:11968 [sender.py:send():380] send: final
2023-11-18 11:40:30,590 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: defer
2023-11-18 11:40:30,591 DEBUG SenderThread:11968 [sender.py:send():380] send: footer
2023-11-18 11:40:30,591 INFO HandlerThread:11968 [handler.py:handle_request_defer():172] handle defer: 14
2023-11-18 11:40:30,592 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: defer
2023-11-18 11:40:30,592 INFO SenderThread:11968 [sender.py:send_request_defer():609] handle sender defer: 14
2023-11-18 11:40:30,593 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: poll_exit
2023-11-18 11:40:30,593 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: poll_exit
2023-11-18 11:40:30,593 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: poll_exit
2023-11-18 11:40:30,594 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: server_info
2023-11-18 11:40:30,594 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: poll_exit
2023-11-18 11:40:30,594 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: get_summary
2023-11-18 11:40:30,595 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: server_info
2023-11-18 11:40:30,595 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: sampled_history
2023-11-18 11:40:30,597 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: internal_messages
2023-11-18 11:40:30,597 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: job_info
2023-11-18 11:40:30,741 DEBUG SenderThread:11968 [sender.py:send_request():407] send_request: job_info
2023-11-18 11:40:30,741 INFO MainThread:11968 [wandb_run.py:_footer_history_summary_info():3760] rendering history
2023-11-18 11:40:30,741 INFO MainThread:11968 [wandb_run.py:_footer_history_summary_info():3792] rendering summary
2023-11-18 11:40:30,742 INFO MainThread:11968 [wandb_run.py:_footer_sync_info():3719] logging synced files
2023-11-18 11:40:30,742 DEBUG HandlerThread:11968 [handler.py:handle_request():146] handle_request: shutdown
2023-11-18 11:40:30,743 INFO HandlerThread:11968 [handler.py:finish():866] shutting down handler
2023-11-18 11:40:31,598 INFO WriterThread:11968 [datastore.py:close():294] close: /mnt/e/wandb/run-20231118_113738-lw67dmi9/run-lw67dmi9.wandb
2023-11-18 11:40:31,741 INFO SenderThread:11968 [sender.py:finish():1536] shutting down sender
2023-11-18 11:40:31,742 INFO SenderThread:11968 [file_pusher.py:finish():175] shutting down file pusher
2023-11-18 11:40:31,742 INFO SenderThread:11968 [file_pusher.py:join():181] waiting for file pusher