File size: 19,464 Bytes
687f89f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
2024-01-07 08:32:15,522 INFO    MainThread:684 [wandb_setup.py:_flush():76] Current SDK version is 0.16.1
2024-01-07 08:32:15,523 INFO    MainThread:684 [wandb_setup.py:_flush():76] Configure stats pid to 684
2024-01-07 08:32:15,523 INFO    MainThread:684 [wandb_setup.py:_flush():76] Loading settings from /root/.config/wandb/settings
2024-01-07 08:32:15,524 INFO    MainThread:684 [wandb_setup.py:_flush():76] Loading settings from /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/settings
2024-01-07 08:32:15,524 INFO    MainThread:684 [wandb_setup.py:_flush():76] Loading settings from environment variables: {}
2024-01-07 08:32:15,524 INFO    MainThread:684 [wandb_setup.py:_flush():76] Applying setup settings: {'_disable_service': False}
2024-01-07 08:32:15,524 INFO    MainThread:684 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program': '<python with no main file>'}
2024-01-07 08:32:15,524 INFO    MainThread:684 [wandb_setup.py:_flush():76] Applying login settings: {'api_key': '***REDACTED***'}
2024-01-07 08:32:15,525 INFO    MainThread:684 [wandb_init.py:_log_setup():524] Logging user logs to /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240107_083215-enryt6zo/logs/debug.log
2024-01-07 08:32:15,525 INFO    MainThread:684 [wandb_init.py:_log_setup():525] Logging internal logs to /content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/wandb/run-20240107_083215-enryt6zo/logs/debug-internal.log
2024-01-07 08:32:15,525 INFO    MainThread:684 [wandb_init.py:_jupyter_setup():470] configuring jupyter hooks <wandb.sdk.wandb_init._WandbInit object at 0x7bfea43750f0>
2024-01-07 08:32:15,526 INFO    MainThread:684 [wandb_init.py:init():564] calling init triggers
2024-01-07 08:32:15,526 INFO    MainThread:684 [wandb_init.py:init():571] wandb.init called with sweep_config: {}
config: {}
2024-01-07 08:32:15,526 INFO    MainThread:684 [wandb_init.py:init():614] starting backend
2024-01-07 08:32:15,526 INFO    MainThread:684 [wandb_init.py:init():618] setting up manager
2024-01-07 08:32:15,531 INFO    MainThread:684 [backend.py:_multiprocessing_setup():105] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
2024-01-07 08:32:15,534 INFO    MainThread:684 [wandb_init.py:init():624] backend started and connected
2024-01-07 08:32:15,570 INFO    MainThread:684 [wandb_run.py:_label_probe_notebook():1294] probe notebook
2024-01-07 08:32:17,418 INFO    MainThread:684 [wandb_init.py:init():716] updated telemetry
2024-01-07 08:32:17,453 INFO    MainThread:684 [wandb_init.py:init():749] communicating run to backend with 90.0 second timeout
2024-01-07 08:32:17,964 INFO    MainThread:684 [wandb_run.py:_on_init():2254] communicating current version
2024-01-07 08:32:18,119 INFO    MainThread:684 [wandb_run.py:_on_init():2263] got version response 
2024-01-07 08:32:18,120 INFO    MainThread:684 [wandb_init.py:init():800] starting run threads in backend
2024-01-07 08:32:18,210 INFO    MainThread:684 [wandb_run.py:_console_start():2233] atexit reg
2024-01-07 08:32:18,211 INFO    MainThread:684 [wandb_run.py:_redirect():2088] redirect: wrap_raw
2024-01-07 08:32:18,211 INFO    MainThread:684 [wandb_run.py:_redirect():2153] Wrapping output streams.
2024-01-07 08:32:18,211 INFO    MainThread:684 [wandb_run.py:_redirect():2178] Redirects installed.
2024-01-07 08:32:18,213 INFO    MainThread:684 [wandb_init.py:init():841] run started, returning control to user process
2024-01-07 08:32:18,219 INFO    MainThread:684 [wandb_run.py:_config_callback():1342] config_cb None None {'vocab_size': 32000, 'max_position_embeddings': 32768, 'hidden_size': 4096, 'intermediate_size': 14336, 'num_hidden_layers': 32, 'num_attention_heads': 32, 'sliding_window': 4096, 'num_key_value_heads': 8, 'hidden_act': 'silu', 'initializer_range': 0.02, 'rms_norm_eps': 1e-05, 'use_cache': False, 'rope_theta': 10000.0, 'attention_dropout': 0.0, 'return_dict': True, 'output_hidden_states': False, 'output_attentions': False, 'torchscript': False, 'torch_dtype': 'bfloat16', 'use_bfloat16': False, 'tf_legacy_loss': False, 'pruned_heads': {}, 'tie_word_embeddings': False, 'is_encoder_decoder': False, 'is_decoder': False, 'cross_attention_hidden_size': None, 'add_cross_attention': False, 'tie_encoder_decoder': False, 'max_length': 20, 'min_length': 0, 'do_sample': False, 'early_stopping': False, 'num_beams': 1, 'num_beam_groups': 1, 'diversity_penalty': 0.0, 'temperature': 1.0, 'top_k': 50, 'top_p': 1.0, 'typical_p': 1.0, 'repetition_penalty': 1.0, 'length_penalty': 1.0, 'no_repeat_ngram_size': 0, 'encoder_no_repeat_ngram_size': 0, 'bad_words_ids': None, 'num_return_sequences': 1, 'chunk_size_feed_forward': 0, 'output_scores': False, 'return_dict_in_generate': False, 'forced_bos_token_id': None, 'forced_eos_token_id': None, 'remove_invalid_values': False, 'exponential_decay_length_penalty': None, 'suppress_tokens': None, 'begin_suppress_tokens': None, 'architectures': ['MistralForCausalLM'], 'finetuning_task': None, 'id2label': {0: 'LABEL_0', 1: 'LABEL_1'}, 'label2id': {'LABEL_0': 0, 'LABEL_1': 1}, 'tokenizer_class': None, 'prefix': None, 'bos_token_id': 1, 'pad_token_id': None, 'eos_token_id': 2, 'sep_token_id': None, 'decoder_start_token_id': None, 'task_specific_params': None, 'problem_type': None, '_name_or_path': 'mistralai/Mistral-7B-v0.1', 'transformers_version': '4.36.2', 'model_type': 'mistral', 'quantization_config': {'quant_method': 'QuantizationMethod.BITS_AND_BYTES', 'load_in_8bit': False, 'load_in_4bit': True, 'llm_int8_threshold': 6.0, 'llm_int8_skip_modules': None, 'llm_int8_enable_fp32_cpu_offload': False, 'llm_int8_has_fp16_weight': False, 'bnb_4bit_quant_type': 'nf4', 'bnb_4bit_use_double_quant': True, 'bnb_4bit_compute_dtype': 'bfloat16'}, 'output_dir': '/content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance', 'overwrite_output_dir': False, 'do_train': False, 'do_eval': False, 'do_predict': False, 'evaluation_strategy': 'no', 'prediction_loss_only': False, 'per_device_train_batch_size': 2, 'per_device_eval_batch_size': 8, 'per_gpu_train_batch_size': None, 'per_gpu_eval_batch_size': None, 'gradient_accumulation_steps': 2, 'eval_accumulation_steps': None, 'eval_delay': 0, 'learning_rate': 0.0002, 'weight_decay': 0.0, 'adam_beta1': 0.9, 'adam_beta2': 0.999, 'adam_epsilon': 1e-08, 'max_grad_norm': 0.3, 'num_train_epochs': 3.0, 'max_steps': 60, 'lr_scheduler_type': 'cosine', 'lr_scheduler_kwargs': {}, 'warmup_ratio': 0.03, 'warmup_steps': 0, 'log_level': 'passive', 'log_level_replica': 'warning', 'log_on_each_node': True, 'logging_dir': '/content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance/runs/Jan07_08-30-52_096ae31a5012', 'logging_strategy': 'steps', 'logging_first_step': False, 'logging_steps': 10, 'logging_nan_inf_filter': True, 'save_strategy': 'steps', 'save_steps': 10, 'save_total_limit': None, 'save_safetensors': True, 'save_on_each_node': False, 'save_only_model': False, 'no_cuda': False, 'use_cpu': False, 'use_mps_device': False, 'seed': 42, 'data_seed': None, 'jit_mode_eval': False, 'use_ipex': False, 'bf16': False, 'fp16': False, 'fp16_opt_level': 'O1', 'half_precision_backend': 'auto', 'bf16_full_eval': False, 'fp16_full_eval': False, 'tf32': False, 'local_rank': 0, 'ddp_backend': None, 'tpu_num_cores': None, 'tpu_metrics_debug': False, 'debug': [], 'dataloader_drop_last': False, 'eval_steps': None, 'dataloader_num_workers': 0, 'past_index': -1, 'run_name': '/content/gdrive/MyDrive/LLM/Mistral-7B-Finetuning-Insurance', 'disable_tqdm': False, 'remove_unused_columns': True, 'label_names': None, 'load_best_model_at_end': False, 'metric_for_best_model': None, 'greater_is_better': None, 'ignore_data_skip': False, 'fsdp': [], 'fsdp_min_num_params': 0, 'fsdp_config': {'min_num_params': 0, 'xla': False, 'xla_fsdp_grad_ckpt': False}, 'fsdp_transformer_layer_cls_to_wrap': None, 'deepspeed': None, 'label_smoothing_factor': 0.0, 'optim': 'paged_adamw_32bit', 'optim_args': None, 'adafactor': False, 'group_by_length': True, 'length_column_name': 'length', 'report_to': ['tensorboard', 'wandb'], 'ddp_find_unused_parameters': None, 'ddp_bucket_cap_mb': None, 'ddp_broadcast_buffers': None, 'dataloader_pin_memory': True, 'dataloader_persistent_workers': False, 'skip_memory_metrics': True, 'use_legacy_prediction_loop': False, 'push_to_hub': True, 'resume_from_checkpoint': None, 'hub_model_id': None, 'hub_strategy': 'every_save', 'hub_token': '<HUB_TOKEN>', 'hub_private_repo': False, 'hub_always_push': False, 'gradient_checkpointing': False, 'gradient_checkpointing_kwargs': None, 'include_inputs_for_metrics': False, 'fp16_backend': 'auto', 'push_to_hub_model_id': None, 'push_to_hub_organization': None, 'push_to_hub_token': '<PUSH_TO_HUB_TOKEN>', 'mp_parameters': '', 'auto_find_batch_size': False, 'full_determinism': False, 'torchdynamo': None, 'ray_scope': 'last', 'ddp_timeout': 1800, 'torch_compile': False, 'torch_compile_backend': None, 'torch_compile_mode': None, 'dispatch_batches': None, 'split_batches': False, 'include_tokens_per_second': False, 'include_num_input_tokens_seen': False, 'neftune_noise_alpha': None}
2024-01-07 08:44:03,889 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 08:44:03,890 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 08:44:26,570 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 08:44:34,326 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 08:44:34,327 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 08:44:46,475 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 08:46:05,058 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 08:46:05,058 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 08:46:13,038 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 08:46:18,516 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 08:46:18,516 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 08:50:09,111 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 08:50:13,508 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 08:50:13,513 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 08:51:38,094 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 08:51:38,098 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 08:51:38,098 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 08:51:41,383 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 08:52:12,662 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 08:52:12,662 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 08:52:45,454 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 08:53:09,095 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 08:53:09,096 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 08:55:59,367 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 08:56:00,251 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 08:56:00,252 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 08:56:06,562 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 08:56:10,699 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 08:56:10,700 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 08:57:55,151 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 08:57:58,981 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 08:57:58,981 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 09:00:08,883 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 09:00:18,822 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 09:00:18,822 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 09:08:01,727 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 09:08:06,231 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 09:08:06,232 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 09:08:25,862 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 09:08:25,898 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 09:08:25,898 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 09:08:35,845 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 09:08:37,716 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 09:08:37,717 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 09:08:40,487 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 09:08:40,495 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 09:08:40,501 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 09:08:45,788 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 09:08:45,793 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 09:08:45,794 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 09:08:49,111 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 09:08:49,155 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 09:08:49,155 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 09:09:44,376 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 09:09:44,379 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 09:09:44,380 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 09:10:16,380 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 09:10:16,383 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 09:10:16,383 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 09:10:25,980 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 09:10:26,068 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 09:10:26,076 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 09:10:52,944 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 09:10:52,950 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 09:10:52,950 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 09:10:54,782 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 09:10:54,813 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 09:10:54,813 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 09:12:03,682 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 09:12:03,692 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 09:12:03,692 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 09:12:06,232 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 09:12:06,325 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 09:12:06,326 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 09:12:33,934 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 09:12:34,001 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 09:12:34,004 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 09:13:00,605 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 09:13:00,639 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 09:13:00,639 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 09:13:06,384 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 09:13:06,462 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 09:13:06,462 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 09:13:24,618 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 09:13:24,664 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 09:13:24,665 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 09:13:52,451 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 09:13:52,481 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 09:13:52,481 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 09:14:01,980 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 09:14:02,039 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 09:14:02,040 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 09:14:37,393 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 09:14:42,402 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 09:14:42,403 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 09:16:27,093 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 09:16:27,127 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 09:16:27,127 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend
2024-01-07 09:19:48,148 INFO    MainThread:684 [wandb_init.py:_resume_backend():440] resuming backend
2024-01-07 09:19:48,193 INFO    MainThread:684 [jupyter.py:save_ipynb():373] not saving jupyter notebook
2024-01-07 09:19:48,193 INFO    MainThread:684 [wandb_init.py:_pause_backend():435] pausing backend