| 2023-11-18 11:42:08,966 INFO StreamThr :13645 [internal.py:wandb_internal():86] W&B internal server running at pid: 13645, started at: 2023-11-18 11:42:08.962667 | |
| 2023-11-18 11:42:08,970 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: status | |
| 2023-11-18 11:42:08,972 INFO WriterThread:13645 [datastore.py:open_for_write():85] open: /mnt/e/wandb/run-20231118_114208-iofrt503/run-iofrt503.wandb | |
| 2023-11-18 11:42:08,977 DEBUG SenderThread:13645 [sender.py:send():380] send: header | |
| 2023-11-18 11:42:08,978 DEBUG SenderThread:13645 [sender.py:send():380] send: run | |
| 2023-11-18 11:42:09,383 INFO SenderThread:13645 [dir_watcher.py:__init__():211] watching files in: /mnt/e/wandb/run-20231118_114208-iofrt503/files | |
| 2023-11-18 11:42:09,392 INFO SenderThread:13645 [sender.py:_start_run_threads():1124] run started: iofrt503 with start time 1700300528.952379 | |
| 2023-11-18 11:42:09,393 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: check_version | |
| 2023-11-18 11:42:09,394 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: check_version | |
| 2023-11-18 11:42:09,547 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: run_start | |
| 2023-11-18 11:42:09,556 DEBUG HandlerThread:13645 [system_info.py:__init__():32] System info init | |
| 2023-11-18 11:42:09,556 DEBUG HandlerThread:13645 [system_info.py:__init__():47] System info init done | |
| 2023-11-18 11:42:09,556 INFO HandlerThread:13645 [system_monitor.py:start():194] Starting system monitor | |
| 2023-11-18 11:42:09,557 INFO SystemMonitor:13645 [system_monitor.py:_start():158] Starting system asset monitoring threads | |
| 2023-11-18 11:42:09,557 INFO HandlerThread:13645 [system_monitor.py:probe():214] Collecting system info | |
| 2023-11-18 11:42:09,558 INFO SystemMonitor:13645 [interfaces.py:start():190] Started cpu monitoring | |
| 2023-11-18 11:42:09,559 INFO SystemMonitor:13645 [interfaces.py:start():190] Started disk monitoring | |
| 2023-11-18 11:42:09,561 INFO SystemMonitor:13645 [interfaces.py:start():190] Started gpu monitoring | |
| 2023-11-18 11:42:09,562 INFO SystemMonitor:13645 [interfaces.py:start():190] Started memory monitoring | |
| 2023-11-18 11:42:09,562 INFO SystemMonitor:13645 [interfaces.py:start():190] Started network monitoring | |
| 2023-11-18 11:42:09,573 DEBUG HandlerThread:13645 [system_info.py:probe():196] Probing system | |
| 2023-11-18 11:42:09,658 DEBUG HandlerThread:13645 [gitlib.py:_init_repo():56] git repository is invalid | |
| 2023-11-18 11:42:09,658 DEBUG HandlerThread:13645 [system_info.py:probe():244] Probing system done | |
| 2023-11-18 11:42:09,659 DEBUG HandlerThread:13645 [system_monitor.py:probe():223] {'os': 'Linux-5.15.123.1-microsoft-standard-WSL2-x86_64-with-glibc2.35', 'python': '3.10.13', 'heartbeatAt': '2023-11-18T09:42:09.573567', 'startedAt': '2023-11-18T09:42:08.901047', 'docker': None, 'cuda': None, 'args': ('--deepspeed=ds_config.json', '--model_name_or_path=openai/whisper-large-v2', '--train_dataset_names', 'mozilla-foundation/common_voice_11_0', 'mozilla-foundation/common_voice_11_0', 'google/fleurs', 'Finnish-NLP/css10_audio_processed', 'Finnish-NLP/youtube_fi_110k_preprocessed', 'Finnish-NLP/voxpopuli_fi_audio_processed', 'Finnish-NLP/parliament_audio_processed', 'Finnish-NLP/aalto_eduskunta_asr_audio_processed', 'Finnish-NLP/fbc_dialog_audio_processed', 'Finnish-NLP/fbc_monolog_audio_processed', '--train_dataset_config_names', 'fi', 'fi', 'fi_fi', '', '', '', '', '', '', '', '--train_dataset_split_name', 'train', 'validation', 'train', 'train', 'train', 'train', 'train', 'train', 'train', 'train', '--train_dataset_text_column_names', 'sentence', 'sentence', 'raw_transcription', 'sentence', 'texts', 'sentence', 'sentence', 'sentence', 'sentence', 'sentence', '--eval_dataset_name=mozilla-foundation/common_voice_11_0', '--eval_dataset_config_name=fi', '--eval_split_name=test', '--language=finnish', '--model_index_name=Whisper Large V3 Finnish', '--max_steps=30000', '--output_dir=./', '--per_device_train_batch_size=8', '--per_device_eval_batch_size=4', '--gradient_accumulation_steps=4', '--logging_steps=25', '--learning_rate=3e-6', '--warmup_steps=1000', '--evaluation_strategy=steps', '--eval_steps=1000', '--save_strategy=steps', '--save_steps=1000', '--generation_max_length=225', '--length_column_name=input_length', '--max_duration_in_seconds=30', '--text_column_name=sentence', '--freeze_feature_encoder=False', '--report_to=wandb', '--metric_for_best_model=wer', '--greater_is_better=False', '--load_best_model_at_end', '--gradient_checkpointing', '--fp16', '--overwrite_output_dir', '--do_train', '--do_eval', '--predict_with_generate', '--do_normalize_eval', '--use_auth_token', '--push_to_hub'), 'state': 'running', 'program': '/mnt/e/run_speech_recognition_seq2seq_streaming.py', 'codePathLocal': 'run_speech_recognition_seq2seq_streaming.py', 'codePath': 'run_speech_recognition_seq2seq_streaming.py', 'host': 'DESKTOP-59O9VN1', 'username': 'rasmus', 'executable': '/home/rasmus/miniconda3/envs/WhisperFinetuneEnv/bin/python', 'cpu_count': 4, 'cpu_count_logical': 8, 'cpu_freq': {'current': 3911.9949999999994, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}, {'current': 3911.995, 'min': 0.0, 'max': 0.0}], 'disk': {'/': {'total': 1006.853931427002, 'used': 37.273990631103516}}, 'gpu': 'NVIDIA GeForce RTX 4080', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA GeForce RTX 4080', 'memory_total': 17171480576}], 'memory': {'total': 54.92573928833008}} | |
| 2023-11-18 11:42:09,659 INFO HandlerThread:13645 [system_monitor.py:probe():224] Finished collecting system info | |
| 2023-11-18 11:42:09,659 INFO HandlerThread:13645 [system_monitor.py:probe():227] Publishing system info | |
| 2023-11-18 11:42:09,659 DEBUG HandlerThread:13645 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment | |
| 2023-11-18 11:42:09,662 DEBUG HandlerThread:13645 [system_info.py:_save_pip():68] Saving pip packages done | |
| 2023-11-18 11:42:09,662 DEBUG HandlerThread:13645 [system_info.py:_save_conda():75] Saving list of conda packages installed into the current environment | |
| 2023-11-18 11:42:10,387 INFO Thread-12 :13645 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_114208-iofrt503/files/requirements.txt | |
| 2023-11-18 11:42:10,388 INFO Thread-12 :13645 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_114208-iofrt503/files/conda-environment.yaml | |
| 2023-11-18 11:42:11,825 DEBUG HandlerThread:13645 [system_info.py:_save_conda():87] Saving conda packages done | |
| 2023-11-18 11:42:11,829 INFO HandlerThread:13645 [system_monitor.py:probe():229] Finished publishing system info | |
| 2023-11-18 11:42:11,832 DEBUG SenderThread:13645 [sender.py:send():380] send: files | |
| 2023-11-18 11:42:11,832 INFO SenderThread:13645 [sender.py:_save_file():1380] saving file wandb-metadata.json with policy now | |
| 2023-11-18 11:42:11,841 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 11:42:11,841 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 11:42:11,842 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 11:42:12,031 DEBUG SenderThread:13645 [sender.py:send():380] send: telemetry | |
| 2023-11-18 11:42:12,031 DEBUG SenderThread:13645 [sender.py:send():380] send: config | |
| 2023-11-18 11:42:12,032 DEBUG SenderThread:13645 [sender.py:send():380] send: metric | |
| 2023-11-18 11:42:12,032 DEBUG SenderThread:13645 [sender.py:send():380] send: telemetry | |
| 2023-11-18 11:42:12,032 DEBUG SenderThread:13645 [sender.py:send():380] send: metric | |
| 2023-11-18 11:42:12,033 WARNING SenderThread:13645 [sender.py:send_metric():1331] Seen metric with glob (shouldn't happen) | |
| 2023-11-18 11:42:12,033 DEBUG SenderThread:13645 [sender.py:send():380] send: telemetry | |
| 2023-11-18 11:42:12,399 INFO Thread-12 :13645 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114208-iofrt503/files/conda-environment.yaml | |
| 2023-11-18 11:42:12,400 INFO Thread-12 :13645 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_114208-iofrt503/files/wandb-metadata.json | |
| 2023-11-18 11:42:12,402 INFO Thread-12 :13645 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_114208-iofrt503/files/output.log | |
| 2023-11-18 11:42:12,438 INFO wandb-upload_0:13645 [upload_job.py:push():131] Uploaded file /tmp/tmpcostdywiwandb/1tczprur-wandb-metadata.json | |
| 2023-11-18 11:42:14,150 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:42:14,410 INFO Thread-12 :13645 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114208-iofrt503/files/output.log | |
| 2023-11-18 11:42:16,429 INFO Thread-12 :13645 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114208-iofrt503/files/output.log | |
| 2023-11-18 11:42:19,311 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:42:24,312 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:42:26,840 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 11:42:26,840 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 11:42:26,842 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 11:42:30,022 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:42:35,023 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:42:40,027 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:42:40,566 INFO Thread-12 :13645 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114208-iofrt503/files/config.yaml | |
| 2023-11-18 11:42:41,840 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 11:42:41,841 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 11:42:41,885 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 11:42:46,043 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:42:51,045 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:42:56,046 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:42:56,840 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 11:42:56,841 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 11:42:56,885 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 11:43:01,067 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:43:06,068 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:43:09,563 DEBUG SystemMonitor:13645 [system_monitor.py:_start():172] Starting system metrics aggregation loop | |
| 2023-11-18 11:43:09,565 DEBUG SenderThread:13645 [sender.py:send():380] send: stats | |
| 2023-11-18 11:43:11,566 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:43:11,840 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 11:43:11,841 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 11:43:11,885 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 11:43:17,026 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:43:22,027 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:43:26,841 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 11:43:26,842 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 11:43:26,885 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 11:43:27,159 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:43:32,160 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:43:37,161 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:43:39,566 DEBUG SenderThread:13645 [sender.py:send():380] send: stats | |
| 2023-11-18 11:43:41,840 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 11:43:41,841 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 11:43:41,885 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 11:43:43,059 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:43:48,070 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:43:53,071 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:43:56,841 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 11:43:56,842 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 11:43:56,885 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 11:43:58,105 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:44:03,106 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:44:08,107 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:44:09,566 DEBUG SenderThread:13645 [sender.py:send():380] send: stats | |
| 2023-11-18 11:44:11,841 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: stop_status | |
| 2023-11-18 11:44:11,841 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: stop_status | |
| 2023-11-18 11:44:11,842 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 11:44:13,023 INFO Thread-12 :13645 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114208-iofrt503/files/output.log | |
| 2023-11-18 11:44:13,119 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:44:17,048 INFO Thread-12 :13645 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114208-iofrt503/files/output.log | |
| 2023-11-18 11:44:17,519 DEBUG SenderThread:13645 [sender.py:send():380] send: exit | |
| 2023-11-18 11:44:17,520 INFO SenderThread:13645 [sender.py:send_exit():585] handling exit code: 1 | |
| 2023-11-18 11:44:17,520 INFO SenderThread:13645 [sender.py:send_exit():587] handling runtime: 127 | |
| 2023-11-18 11:44:17,531 INFO SenderThread:13645 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end | |
| 2023-11-18 11:44:17,532 INFO SenderThread:13645 [sender.py:send_exit():593] send defer | |
| 2023-11-18 11:44:17,532 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:44:17,532 INFO HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 0 | |
| 2023-11-18 11:44:17,532 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:44:17,533 INFO SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 0 | |
| 2023-11-18 11:44:17,533 INFO SenderThread:13645 [sender.py:transition_state():613] send defer: 1 | |
| 2023-11-18 11:44:17,533 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:44:17,533 INFO HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 1 | |
| 2023-11-18 11:44:17,534 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:44:17,534 INFO SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 1 | |
| 2023-11-18 11:44:17,534 INFO SenderThread:13645 [sender.py:transition_state():613] send defer: 2 | |
| 2023-11-18 11:44:17,534 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:44:17,535 INFO HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 2 | |
| 2023-11-18 11:44:17,535 INFO HandlerThread:13645 [system_monitor.py:finish():203] Stopping system monitor | |
| 2023-11-18 11:44:17,535 DEBUG SystemMonitor:13645 [system_monitor.py:_start():179] Finished system metrics aggregation loop | |
| 2023-11-18 11:44:17,536 DEBUG SystemMonitor:13645 [system_monitor.py:_start():183] Publishing last batch of metrics | |
| 2023-11-18 11:44:17,536 INFO HandlerThread:13645 [interfaces.py:finish():202] Joined cpu monitor | |
| 2023-11-18 11:44:17,537 INFO HandlerThread:13645 [interfaces.py:finish():202] Joined disk monitor | |
| 2023-11-18 11:44:17,548 INFO HandlerThread:13645 [interfaces.py:finish():202] Joined gpu monitor | |
| 2023-11-18 11:44:17,549 INFO HandlerThread:13645 [interfaces.py:finish():202] Joined memory monitor | |
| 2023-11-18 11:44:17,549 INFO HandlerThread:13645 [interfaces.py:finish():202] Joined network monitor | |
| 2023-11-18 11:44:17,550 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:44:17,550 INFO SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 2 | |
| 2023-11-18 11:44:17,550 INFO SenderThread:13645 [sender.py:transition_state():613] send defer: 3 | |
| 2023-11-18 11:44:17,550 DEBUG SenderThread:13645 [sender.py:send():380] send: stats | |
| 2023-11-18 11:44:17,551 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:44:17,551 INFO HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 3 | |
| 2023-11-18 11:44:17,551 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:44:17,552 INFO SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 3 | |
| 2023-11-18 11:44:17,552 INFO SenderThread:13645 [sender.py:transition_state():613] send defer: 4 | |
| 2023-11-18 11:44:17,552 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:44:17,552 INFO HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 4 | |
| 2023-11-18 11:44:17,553 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:44:17,553 INFO SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 4 | |
| 2023-11-18 11:44:17,553 INFO SenderThread:13645 [sender.py:transition_state():613] send defer: 5 | |
| 2023-11-18 11:44:17,553 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:44:17,553 INFO HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 5 | |
| 2023-11-18 11:44:17,554 DEBUG SenderThread:13645 [sender.py:send():380] send: summary | |
| 2023-11-18 11:44:17,556 INFO SenderThread:13645 [sender.py:_save_file():1380] saving file wandb-summary.json with policy end | |
| 2023-11-18 11:44:17,557 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:44:17,557 INFO SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 5 | |
| 2023-11-18 11:44:17,557 INFO SenderThread:13645 [sender.py:transition_state():613] send defer: 6 | |
| 2023-11-18 11:44:17,557 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:44:17,557 INFO HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 6 | |
| 2023-11-18 11:44:17,558 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:44:17,558 INFO SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 6 | |
| 2023-11-18 11:44:17,558 INFO SenderThread:13645 [sender.py:transition_state():613] send defer: 7 | |
| 2023-11-18 11:44:17,558 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: status_report | |
| 2023-11-18 11:44:17,559 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:44:17,559 INFO HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 7 | |
| 2023-11-18 11:44:17,559 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:44:17,559 INFO SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 7 | |
| 2023-11-18 11:44:18,055 INFO Thread-12 :13645 [dir_watcher.py:_on_file_created():271] file/dir created: /mnt/e/wandb/run-20231118_114208-iofrt503/files/wandb-summary.json | |
| 2023-11-18 11:44:18,183 INFO SenderThread:13645 [sender.py:transition_state():613] send defer: 8 | |
| 2023-11-18 11:44:18,184 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:44:18,184 INFO HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 8 | |
| 2023-11-18 11:44:18,184 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:44:18,185 INFO SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 8 | |
| 2023-11-18 11:44:18,185 INFO SenderThread:13645 [job_builder.py:build():298] Attempting to build job artifact | |
| 2023-11-18 11:44:18,191 INFO SenderThread:13645 [job_builder.py:_get_source_type():439] no source found | |
| 2023-11-18 11:44:18,191 INFO SenderThread:13645 [sender.py:transition_state():613] send defer: 9 | |
| 2023-11-18 11:44:18,192 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:44:18,192 INFO HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 9 | |
| 2023-11-18 11:44:18,192 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:44:18,193 INFO SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 9 | |
| 2023-11-18 11:44:18,193 INFO SenderThread:13645 [dir_watcher.py:finish():358] shutting down directory watcher | |
| 2023-11-18 11:44:18,519 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: poll_exit | |
| 2023-11-18 11:44:19,072 INFO SenderThread:13645 [dir_watcher.py:_on_file_modified():288] file/dir modified: /mnt/e/wandb/run-20231118_114208-iofrt503/files/output.log | |
| 2023-11-18 11:44:19,073 INFO SenderThread:13645 [dir_watcher.py:finish():388] scan: /mnt/e/wandb/run-20231118_114208-iofrt503/files | |
| 2023-11-18 11:44:19,075 INFO SenderThread:13645 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_114208-iofrt503/files/conda-environment.yaml conda-environment.yaml | |
| 2023-11-18 11:44:19,078 INFO SenderThread:13645 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_114208-iofrt503/files/config.yaml config.yaml | |
| 2023-11-18 11:44:19,086 INFO SenderThread:13645 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_114208-iofrt503/files/output.log output.log | |
| 2023-11-18 11:44:19,092 INFO SenderThread:13645 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_114208-iofrt503/files/requirements.txt requirements.txt | |
| 2023-11-18 11:44:19,098 INFO SenderThread:13645 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_114208-iofrt503/files/wandb-metadata.json wandb-metadata.json | |
| 2023-11-18 11:44:19,098 INFO SenderThread:13645 [dir_watcher.py:finish():402] scan save: /mnt/e/wandb/run-20231118_114208-iofrt503/files/wandb-summary.json wandb-summary.json | |
| 2023-11-18 11:44:19,103 INFO SenderThread:13645 [sender.py:transition_state():613] send defer: 10 | |
| 2023-11-18 11:44:19,104 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: poll_exit | |
| 2023-11-18 11:44:19,104 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:44:19,105 INFO HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 10 | |
| 2023-11-18 11:44:19,108 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:44:19,108 INFO SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 10 | |
| 2023-11-18 11:44:19,108 INFO SenderThread:13645 [file_pusher.py:finish():175] shutting down file pusher | |
| 2023-11-18 11:44:19,496 INFO wandb-upload_0:13645 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_114208-iofrt503/files/conda-environment.yaml | |
| 2023-11-18 11:44:19,520 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: poll_exit | |
| 2023-11-18 11:44:19,521 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: poll_exit | |
| 2023-11-18 11:44:19,572 INFO wandb-upload_1:13645 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_114208-iofrt503/files/config.yaml | |
| 2023-11-18 11:44:19,612 INFO wandb-upload_4:13645 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_114208-iofrt503/files/wandb-summary.json | |
| 2023-11-18 11:44:19,633 INFO wandb-upload_2:13645 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_114208-iofrt503/files/output.log | |
| 2023-11-18 11:44:19,642 INFO wandb-upload_3:13645 [upload_job.py:push():131] Uploaded file /mnt/e/wandb/run-20231118_114208-iofrt503/files/requirements.txt | |
| 2023-11-18 11:44:19,843 INFO Thread-11 (_thread_body):13645 [sender.py:transition_state():613] send defer: 11 | |
| 2023-11-18 11:44:19,844 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:44:19,844 INFO HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 11 | |
| 2023-11-18 11:44:19,845 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:44:19,845 INFO SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 11 | |
| 2023-11-18 11:44:19,845 INFO SenderThread:13645 [file_pusher.py:join():181] waiting for file pusher | |
| 2023-11-18 11:44:19,846 INFO SenderThread:13645 [sender.py:transition_state():613] send defer: 12 | |
| 2023-11-18 11:44:19,846 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:44:19,846 INFO HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 12 | |
| 2023-11-18 11:44:19,847 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:44:19,847 INFO SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 12 | |
| 2023-11-18 11:44:19,847 INFO SenderThread:13645 [file_stream.py:finish():595] file stream finish called | |
| 2023-11-18 11:44:20,197 INFO SenderThread:13645 [file_stream.py:finish():599] file stream finish is done | |
| 2023-11-18 11:44:20,197 INFO SenderThread:13645 [sender.py:transition_state():613] send defer: 13 | |
| 2023-11-18 11:44:20,198 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:44:20,198 INFO HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 13 | |
| 2023-11-18 11:44:20,199 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:44:20,199 INFO SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 13 | |
| 2023-11-18 11:44:20,199 INFO SenderThread:13645 [sender.py:transition_state():613] send defer: 14 | |
| 2023-11-18 11:44:20,200 DEBUG SenderThread:13645 [sender.py:send():380] send: final | |
| 2023-11-18 11:44:20,200 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: defer | |
| 2023-11-18 11:44:20,200 DEBUG SenderThread:13645 [sender.py:send():380] send: footer | |
| 2023-11-18 11:44:20,201 INFO HandlerThread:13645 [handler.py:handle_request_defer():172] handle defer: 14 | |
| 2023-11-18 11:44:20,202 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: defer | |
| 2023-11-18 11:44:20,202 INFO SenderThread:13645 [sender.py:send_request_defer():609] handle sender defer: 14 | |
| 2023-11-18 11:44:20,203 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: poll_exit | |
| 2023-11-18 11:44:20,203 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: poll_exit | |
| 2023-11-18 11:44:20,204 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: poll_exit | |
| 2023-11-18 11:44:20,204 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: server_info | |
| 2023-11-18 11:44:20,204 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: poll_exit | |
| 2023-11-18 11:44:20,205 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: get_summary | |
| 2023-11-18 11:44:20,205 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: server_info | |
| 2023-11-18 11:44:20,206 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: sampled_history | |
| 2023-11-18 11:44:20,208 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: internal_messages | |
| 2023-11-18 11:44:20,209 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: job_info | |
| 2023-11-18 11:44:20,371 DEBUG SenderThread:13645 [sender.py:send_request():407] send_request: job_info | |
| 2023-11-18 11:44:20,371 INFO MainThread:13645 [wandb_run.py:_footer_history_summary_info():3760] rendering history | |
| 2023-11-18 11:44:20,372 INFO MainThread:13645 [wandb_run.py:_footer_history_summary_info():3792] rendering summary | |
| 2023-11-18 11:44:20,372 INFO MainThread:13645 [wandb_run.py:_footer_sync_info():3719] logging synced files | |
| 2023-11-18 11:44:20,372 DEBUG HandlerThread:13645 [handler.py:handle_request():146] handle_request: shutdown | |
| 2023-11-18 11:44:20,372 INFO HandlerThread:13645 [handler.py:finish():866] shutting down handler | |
| 2023-11-18 11:44:21,209 INFO WriterThread:13645 [datastore.py:close():294] close: /mnt/e/wandb/run-20231118_114208-iofrt503/run-iofrt503.wandb | |
| 2023-11-18 11:44:21,371 INFO SenderThread:13645 [sender.py:finish():1536] shutting down sender | |
| 2023-11-18 11:44:21,372 INFO SenderThread:13645 [file_pusher.py:finish():175] shutting down file pusher | |
| 2023-11-18 11:44:21,372 INFO SenderThread:13645 [file_pusher.py:join():181] waiting for file pusher | |