|
2024-02-27 15:10:59,120 - INFO: Problem Type: text_causal_classification_modeling |
|
2024-02-27 15:10:59,120 - INFO: Global random seed: 550978 |
|
2024-02-27 15:10:59,120 - INFO: Preparing the data... |
|
2024-02-27 15:10:59,120 - INFO: Setting up automatic validation split... |
|
2024-02-27 15:10:59,161 - INFO: Preparing train and validation data |
|
2024-02-27 15:10:59,161 - INFO: Loading train dataset... |
|
2024-02-27 15:10:59,884 - INFO: Loading validation dataset... |
|
2024-02-27 15:11:00,030 - INFO: Number of observations in train dataset: 10800 |
|
2024-02-27 15:11:00,030 - INFO: Number of observations in validation dataset: 1200 |
|
2024-02-27 15:11:00,338 - WARNING: |
|
WARNING: You are currently loading Falcon using legacy code contained in the model repository. Falcon has now been fully ported into the Hugging Face transformers library. For the most up-to-date and high-performance version of the Falcon model code, please update to the latest version of transformers and then load the model without the trust_remote_code=True argument. |
|
|
|
2024-02-27 15:11:00,476 - WARNING: PAD token id not matching between config and tokenizer. Overwriting with tokenizer id. |
|
2024-02-27 15:11:00,483 - INFO: Using bfloat16 for backbone |
|
2024-02-27 15:11:00,483 - INFO: Loading tiiuae/falcon-7b. This may take a while. |
|
2024-02-27 15:11:48,531 - INFO: Loaded tiiuae/falcon-7b. |
|
2024-02-27 15:11:48,533 - WARNING: PAD token id not matching between generation config and tokenizer. Overwriting with tokenizer id. |
|
2024-02-27 15:11:48,533 - INFO: Lora module names: ['query_key_value', 'dense', 'dense_h_to_4h', 'dense_4h_to_h'] |
|
2024-02-27 15:11:48,673 - INFO: Enough space available for saving model weights.Required space: 13632.81MB, Available space: 991138.32MB. |
|
2024-02-27 15:11:48,818 - INFO: Training Epoch: 1 / 15 |
|
2024-02-27 15:11:48,818 - INFO: train loss: 0%| | 0/1350 [00:00<?, ?it/s] |
|
2024-02-27 15:11:48,897 - INFO: Evaluation step: 1350 |
|
2024-02-27 15:12:38,005 - INFO: train loss: 5.39: 5%|4 | 67/1350 [00:49<15:41, 1.36it/s] |
|
2024-02-27 15:12:48,819 - INFO: train loss: 5.39: 5%|4 | 67/1350 [01:00<15:41, 1.36it/s] |
|
2024-02-27 15:13:27,421 - INFO: train loss: 4.75: 10%|9 | 134/1350 [01:38<14:55, 1.36it/s] |
|
2024-02-27 15:13:38,821 - INFO: train loss: 4.75: 10%|9 | 134/1350 [01:50<14:55, 1.36it/s] |
|
2024-02-27 15:14:16,304 - INFO: train loss: 4.33: 15%|#4 | 201/1350 [02:27<14:02, 1.36it/s] |
|
2024-02-27 15:14:28,822 - INFO: train loss: 4.33: 15%|#4 | 201/1350 [02:40<14:02, 1.36it/s] |
|
2024-02-27 15:15:04,428 - INFO: train loss: 2.26: 20%|#9 | 268/1350 [03:15<13:06, 1.37it/s] |
|
2024-02-27 15:15:18,823 - INFO: train loss: 2.26: 20%|#9 | 268/1350 [03:30<13:06, 1.37it/s] |
|
2024-02-27 15:15:52,987 - INFO: train loss: 1.87: 25%|##4 | 335/1350 [04:04<12:17, 1.38it/s] |
|
2024-02-27 15:16:08,825 - INFO: train loss: 1.87: 25%|##4 | 335/1350 [04:20<12:17, 1.38it/s] |
|
2024-02-27 15:16:40,862 - INFO: train loss: 2.10: 30%|##9 | 402/1350 [04:52<11:24, 1.38it/s] |
|
2024-02-27 15:16:58,826 - INFO: train loss: 2.10: 30%|##9 | 402/1350 [05:10<11:24, 1.38it/s] |
|
2024-02-27 15:17:29,975 - INFO: train loss: 1.72: 35%|###4 | 469/1350 [05:41<10:39, 1.38it/s] |
|
2024-02-27 15:17:40,723 - INFO: train loss: 1.72: 35%|###4 | 469/1350 [05:51<10:39, 1.38it/s] |
|
2024-02-27 15:18:18,537 - INFO: train loss: 1.97: 40%|###9 | 536/1350 [06:29<09:50, 1.38it/s] |
|
2024-02-27 15:18:28,828 - INFO: train loss: 1.97: 40%|###9 | 536/1350 [06:40<09:50, 1.38it/s] |
|
2024-02-27 15:19:07,071 - INFO: train loss: 1.56: 45%|####4 | 603/1350 [07:18<09:01, 1.38it/s] |
|
2024-02-27 15:19:18,829 - INFO: train loss: 1.56: 45%|####4 | 603/1350 [07:30<09:01, 1.38it/s] |
|
2024-02-27 15:19:56,260 - INFO: train loss: 1.73: 50%|####9 | 670/1350 [08:07<08:14, 1.37it/s] |
|
2024-02-27 15:20:08,830 - INFO: train loss: 1.73: 50%|####9 | 670/1350 [08:20<08:14, 1.37it/s] |
|
2024-02-27 15:20:45,822 - INFO: train loss: 1.89: 55%|#####4 | 737/1350 [08:57<07:28, 1.37it/s] |
|
2024-02-27 15:20:58,832 - INFO: train loss: 1.89: 55%|#####4 | 737/1350 [09:10<07:28, 1.37it/s] |
|
2024-02-27 15:21:34,932 - INFO: train loss: 1.68: 60%|#####9 | 804/1350 [09:46<06:39, 1.37it/s] |
|
2024-02-27 15:21:48,833 - INFO: train loss: 1.68: 60%|#####9 | 804/1350 [10:00<06:39, 1.37it/s] |
|
2024-02-27 15:22:24,373 - INFO: train loss: 1.53: 65%|######4 | 871/1350 [10:35<05:51, 1.36it/s] |
|
2024-02-27 15:22:38,834 - INFO: train loss: 1.53: 65%|######4 | 871/1350 [10:50<05:51, 1.36it/s] |
|
2024-02-27 15:23:11,765 - INFO: train loss: 1.50: 69%|######9 | 938/1350 [11:22<04:59, 1.38it/s] |
|
2024-02-27 15:23:28,835 - INFO: train loss: 1.50: 69%|######9 | 938/1350 [11:40<04:59, 1.38it/s] |
|
2024-02-27 15:24:01,645 - INFO: train loss: 1.55: 74%|#######4 | 1005/1350 [12:12<04:12, 1.37it/s] |
|
2024-02-27 15:24:18,836 - INFO: train loss: 1.55: 74%|#######4 | 1005/1350 [12:30<04:12, 1.37it/s] |
|
2024-02-27 15:24:50,764 - INFO: train loss: 1.31: 79%|#######9 | 1072/1350 [13:01<03:23, 1.37it/s] |
|
2024-02-27 15:25:08,837 - INFO: train loss: 1.31: 79%|#######9 | 1072/1350 [13:20<03:23, 1.37it/s] |
|
2024-02-27 15:25:41,093 - INFO: train loss: 1.22: 84%|########4 | 1139/1350 [13:52<02:35, 1.36it/s] |
|
2024-02-27 15:25:58,839 - INFO: train loss: 1.22: 84%|########4 | 1139/1350 [14:10<02:35, 1.36it/s] |
|
2024-02-27 15:26:30,846 - INFO: train loss: 1.10: 89%|########9 | 1206/1350 [14:42<01:46, 1.35it/s] |
|
2024-02-27 15:26:48,840 - INFO: train loss: 1.10: 89%|########9 | 1206/1350 [15:00<01:46, 1.35it/s] |
|
2024-02-27 15:27:20,319 - INFO: train loss: 1.06: 94%|#########4| 1273/1350 [15:31<00:56, 1.35it/s] |
|
2024-02-27 15:27:30,735 - INFO: train loss: 1.06: 94%|#########4| 1273/1350 [15:41<00:56, 1.35it/s] |
|
2024-02-27 15:28:10,080 - INFO: train loss: 1.05: 99%|#########9| 1340/1350 [16:21<00:07, 1.35it/s] |
|
2024-02-27 15:28:17,574 - INFO: train loss: 1.04: 100%|##########| 1350/1350 [16:28<00:00, 1.37it/s] |
|
2024-02-27 15:28:17,578 - INFO: Starting validation inference |
|
2024-02-27 15:28:17,578 - INFO: validation progress: 0%| | 0/1200 [00:00<?, ?it/s] |
|
2024-02-27 15:28:21,713 - INFO: validation progress: 5%|5 | 60/1200 [00:04<01:18, 14.51it/s] |
|
2024-02-27 15:28:25,795 - INFO: validation progress: 10%|# | 120/1200 [00:08<01:13, 14.62it/s] |
|
2024-02-27 15:28:29,801 - INFO: validation progress: 15%|#5 | 180/1200 [00:12<01:09, 14.78it/s] |
|
2024-02-27 15:28:33,844 - INFO: validation progress: 20%|## | 240/1200 [00:16<01:04, 14.80it/s] |
|
2024-02-27 15:28:37,910 - INFO: validation progress: 25%|##5 | 300/1200 [00:20<01:00, 14.79it/s] |
|
2024-02-27 15:28:41,956 - INFO: validation progress: 30%|### | 360/1200 [00:24<00:56, 14.80it/s] |
|
2024-02-27 15:28:46,043 - INFO: validation progress: 35%|###5 | 420/1200 [00:28<00:52, 14.76it/s] |
|
2024-02-27 15:28:50,095 - INFO: validation progress: 40%|#### | 480/1200 [00:32<00:48, 14.78it/s] |
|
2024-02-27 15:28:54,158 - INFO: validation progress: 45%|####5 | 540/1200 [00:36<00:44, 14.77it/s] |
|
2024-02-27 15:28:58,232 - INFO: validation progress: 50%|##### | 600/1200 [00:40<00:40, 14.76it/s] |
|
2024-02-27 15:29:02,307 - INFO: validation progress: 55%|#####5 | 660/1200 [00:44<00:36, 14.75it/s] |
|
2024-02-27 15:29:06,379 - INFO: validation progress: 60%|###### | 720/1200 [00:48<00:32, 14.74it/s] |
|
2024-02-27 15:29:10,386 - INFO: validation progress: 65%|######5 | 780/1200 [00:52<00:28, 14.81it/s] |
|
2024-02-27 15:29:14,485 - INFO: validation progress: 70%|####### | 840/1200 [00:56<00:24, 14.76it/s] |
|
2024-02-27 15:29:18,557 - INFO: validation progress: 75%|#######5 | 900/1200 [01:00<00:20, 14.75it/s] |
|
2024-02-27 15:29:22,629 - INFO: validation progress: 80%|######## | 960/1200 [01:05<00:16, 14.75it/s] |
|
2024-02-27 15:29:26,707 - INFO: validation progress: 85%|########5 | 1020/1200 [01:09<00:12, 14.74it/s] |
|
2024-02-27 15:29:30,801 - INFO: validation progress: 90%|######### | 1080/1200 [01:13<00:08, 14.71it/s] |
|
2024-02-27 15:29:34,843 - INFO: validation progress: 95%|#########5| 1140/1200 [01:17<00:04, 14.75it/s] |
|
2024-02-27 15:29:38,837 - INFO: validation progress: 100%|##########| 1200/1200 [01:21<00:00, 14.83it/s] |
|
2024-02-27 15:29:38,839 - INFO: validation progress: 100%|##########| 1200/1200 [01:21<00:00, 14.77it/s] |
|
2024-02-27 15:29:38,864 - INFO: Mean validation loss: 0.90334 |
|
2024-02-27 15:29:38,866 - INFO: Validation AUC: 0.79197 |
|
2024-02-27 15:29:38,903 - INFO: Saving best model checkpoint: val_AUC -inf -> 0.79197 to /app/output |
|
2024-02-27 15:29:52,269 - INFO: Training Epoch: 2 / 15 |
|
2024-02-27 15:29:52,270 - INFO: train loss: 0%| | 0/1350 [00:00<?, ?it/s] |
|
2024-02-27 15:29:52,294 - INFO: Evaluation step: 1350 |
|
2024-02-27 15:30:40,267 - INFO: train loss: 0.98: 5%|4 | 67/1350 [00:47<15:19, 1.40it/s] |
|
2024-02-27 15:30:50,846 - INFO: train loss: 0.98: 5%|4 | 67/1350 [00:58<15:19, 1.40it/s] |
|
2024-02-27 15:31:28,420 - INFO: train loss: 0.76: 10%|9 | 134/1350 [01:36<14:32, 1.39it/s] |
|
2024-02-27 15:31:38,853 - INFO: train loss: 0.76: 10%|9 | 134/1350 [01:46<14:32, 1.39it/s] |
|
2024-02-27 15:32:17,535 - INFO: train loss: 0.92: 15%|#4 | 201/1350 [02:25<13:52, 1.38it/s] |
|
2024-02-27 15:32:28,855 - INFO: train loss: 0.92: 15%|#4 | 201/1350 [02:36<13:52, 1.38it/s] |
|
2024-02-27 15:33:07,429 - INFO: train loss: 0.52: 20%|#9 | 268/1350 [03:15<13:12, 1.36it/s] |
|
2024-02-27 15:33:18,856 - INFO: train loss: 0.52: 20%|#9 | 268/1350 [03:26<13:12, 1.36it/s] |
|
2024-02-27 15:33:56,664 - INFO: train loss: 0.59: 25%|##4 | 335/1350 [04:04<12:24, 1.36it/s] |
|
2024-02-27 15:34:08,857 - INFO: train loss: 0.59: 25%|##4 | 335/1350 [04:16<12:24, 1.36it/s] |
|
2024-02-27 15:34:46,245 - INFO: train loss: 0.31: 30%|##9 | 402/1350 [04:53<11:37, 1.36it/s] |
|
2024-02-27 15:34:58,858 - INFO: train loss: 0.31: 30%|##9 | 402/1350 [05:06<11:37, 1.36it/s] |
|
2024-02-27 15:35:36,310 - INFO: train loss: 0.47: 35%|###4 | 469/1350 [05:44<10:51, 1.35it/s] |
|
2024-02-27 15:35:48,860 - INFO: train loss: 0.47: 35%|###4 | 469/1350 [05:56<10:51, 1.35it/s] |
|
2024-02-27 15:36:26,152 - INFO: train loss: 0.66: 40%|###9 | 536/1350 [06:33<10:03, 1.35it/s] |
|
2024-02-27 15:36:38,861 - INFO: train loss: 0.66: 40%|###9 | 536/1350 [06:46<10:03, 1.35it/s] |
|
2024-02-27 15:37:14,471 - INFO: train loss: 0.40: 45%|####4 | 603/1350 [07:22<09:08, 1.36it/s] |
|
2024-02-27 15:37:28,862 - INFO: train loss: 0.40: 45%|####4 | 603/1350 [07:36<09:08, 1.36it/s] |
|
2024-02-27 15:38:02,866 - INFO: train loss: 0.56: 50%|####9 | 670/1350 [08:10<08:17, 1.37it/s] |
|
2024-02-27 15:38:18,864 - INFO: train loss: 0.56: 50%|####9 | 670/1350 [08:26<08:17, 1.37it/s] |
|
2024-02-27 15:38:52,699 - INFO: train loss: 0.33: 55%|#####4 | 737/1350 [09:00<07:30, 1.36it/s] |
|
2024-02-27 15:39:08,865 - INFO: train loss: 0.33: 55%|#####4 | 737/1350 [09:16<07:30, 1.36it/s] |
|
2024-02-27 15:39:42,366 - INFO: train loss: 0.48: 60%|#####9 | 804/1350 [09:50<06:42, 1.36it/s] |
|
2024-02-27 15:39:58,866 - INFO: train loss: 0.48: 60%|#####9 | 804/1350 [10:06<06:42, 1.36it/s] |
|
2024-02-27 15:40:30,326 - INFO: train loss: 0.37: 65%|######4 | 871/1350 [10:38<05:49, 1.37it/s] |
|
2024-02-27 15:40:40,857 - INFO: train loss: 0.37: 65%|######4 | 871/1350 [10:48<05:49, 1.37it/s] |
|
2024-02-27 15:41:19,361 - INFO: train loss: 0.56: 69%|######9 | 938/1350 [11:27<05:01, 1.37it/s] |
|
2024-02-27 15:41:29,499 - INFO: train loss: 0.56: 69%|######9 | 938/1350 [11:37<05:01, 1.37it/s] |
|
2024-02-27 15:42:08,067 - INFO: train loss: 0.32: 74%|#######4 | 1005/1350 [12:15<04:11, 1.37it/s] |
|
2024-02-27 15:42:18,869 - INFO: train loss: 0.32: 74%|#######4 | 1005/1350 [12:26<04:11, 1.37it/s] |
|
2024-02-27 15:42:57,081 - INFO: train loss: 0.57: 79%|#######9 | 1072/1350 [13:04<03:23, 1.37it/s] |
|
2024-02-27 15:43:08,871 - INFO: train loss: 0.57: 79%|#######9 | 1072/1350 [13:16<03:23, 1.37it/s] |
|
2024-02-27 15:43:46,836 - INFO: train loss: 0.39: 84%|########4 | 1139/1350 [13:54<02:34, 1.36it/s] |
|
2024-02-27 15:43:58,872 - INFO: train loss: 0.39: 84%|########4 | 1139/1350 [14:06<02:34, 1.36it/s] |
|
2024-02-27 15:44:34,530 - INFO: train loss: 0.47: 89%|########9 | 1206/1350 [14:42<01:44, 1.37it/s] |
|
2024-02-27 15:44:48,873 - INFO: train loss: 0.47: 89%|########9 | 1206/1350 [14:56<01:44, 1.37it/s] |
|
2024-02-27 15:45:24,188 - INFO: train loss: 0.38: 94%|#########4| 1273/1350 [15:31<00:56, 1.37it/s] |
|
2024-02-27 15:45:38,875 - INFO: train loss: 0.38: 94%|#########4| 1273/1350 [15:46<00:56, 1.37it/s] |
|
2024-02-27 15:46:12,578 - INFO: train loss: 0.26: 99%|#########9| 1340/1350 [16:20<00:07, 1.37it/s] |
|
2024-02-27 15:46:19,467 - INFO: train loss: 0.29: 100%|##########| 1350/1350 [16:27<00:00, 1.37it/s] |
|
2024-02-27 15:46:19,472 - INFO: Starting validation inference |
|
2024-02-27 15:46:19,472 - INFO: validation progress: 0%| | 0/1200 [00:00<?, ?it/s] |
|
2024-02-27 15:46:23,636 - INFO: validation progress: 5%|5 | 60/1200 [00:04<01:19, 14.41it/s] |
|
2024-02-27 15:46:27,760 - INFO: validation progress: 10%|# | 120/1200 [00:08<01:14, 14.49it/s] |
|
2024-02-27 15:46:31,812 - INFO: validation progress: 15%|#5 | 180/1200 [00:12<01:09, 14.63it/s] |
|
2024-02-27 15:46:35,897 - INFO: validation progress: 20%|## | 240/1200 [00:16<01:05, 14.66it/s] |
|
2024-02-27 15:46:40,004 - INFO: validation progress: 25%|##5 | 300/1200 [00:20<01:01, 14.64it/s] |
|
2024-02-27 15:46:44,081 - INFO: validation progress: 30%|### | 360/1200 [00:24<00:57, 14.67it/s] |
|
2024-02-27 15:46:48,203 - INFO: validation progress: 35%|###5 | 420/1200 [00:28<00:53, 14.63it/s] |
|
2024-02-27 15:46:52,276 - INFO: validation progress: 40%|#### | 480/1200 [00:32<00:49, 14.66it/s] |
|
2024-02-27 15:46:56,415 - INFO: validation progress: 45%|####5 | 540/1200 [00:36<00:45, 14.61it/s] |
|
2024-02-27 15:47:00,536 - INFO: validation progress: 50%|##### | 600/1200 [00:41<00:41, 14.59it/s] |
|
2024-02-27 15:47:04,650 - INFO: validation progress: 55%|#####5 | 660/1200 [00:45<00:37, 14.59it/s] |
|
2024-02-27 15:47:08,748 - INFO: validation progress: 60%|###### | 720/1200 [00:49<00:32, 14.61it/s] |
|
2024-02-27 15:47:12,801 - INFO: validation progress: 65%|######5 | 780/1200 [00:53<00:28, 14.67it/s] |
|
2024-02-27 15:47:16,908 - INFO: validation progress: 70%|####### | 840/1200 [00:57<00:24, 14.65it/s] |
|
2024-02-27 15:47:21,016 - INFO: validation progress: 75%|#######5 | 900/1200 [01:01<00:20, 14.64it/s] |
|
2024-02-27 15:47:25,091 - INFO: validation progress: 80%|######## | 960/1200 [01:05<00:16, 14.66it/s] |
|
2024-02-27 15:47:29,177 - INFO: validation progress: 85%|########5 | 1020/1200 [01:09<00:12, 14.67it/s] |
|
2024-02-27 15:47:33,287 - INFO: validation progress: 90%|######### | 1080/1200 [01:13<00:08, 14.65it/s] |
|
2024-02-27 15:47:37,347 - INFO: validation progress: 95%|#########5| 1140/1200 [01:17<00:04, 14.69it/s] |
|
2024-02-27 15:47:41,364 - INFO: validation progress: 100%|##########| 1200/1200 [01:21<00:00, 14.76it/s] |
|
2024-02-27 15:47:41,366 - INFO: validation progress: 100%|##########| 1200/1200 [01:21<00:00, 14.65it/s] |
|
2024-02-27 15:47:41,393 - INFO: Mean validation loss: 0.34724 |
|
2024-02-27 15:47:41,394 - INFO: Validation AUC: 0.97231 |
|
2024-02-27 15:47:41,433 - INFO: Saving best model checkpoint: val_AUC 0.79197 -> 0.97231 to /app/output |
|
2024-02-27 15:47:55,634 - INFO: Training Epoch: 3 / 15 |
|
2024-02-27 15:47:55,634 - INFO: train loss: 0%| | 0/1350 [00:00<?, ?it/s] |
|
2024-02-27 15:47:55,659 - INFO: Evaluation step: 1350 |
|
2024-02-27 15:48:44,330 - INFO: train loss: 0.24: 5%|4 | 67/1350 [00:48<15:32, 1.38it/s] |
|
2024-02-27 15:48:58,962 - INFO: train loss: 0.24: 5%|4 | 67/1350 [01:03<15:32, 1.38it/s] |
|
2024-02-27 15:49:33,947 - INFO: train loss: 0.24: 10%|9 | 134/1350 [01:38<14:53, 1.36it/s] |
|
2024-02-27 15:49:48,963 - INFO: train loss: 0.24: 10%|9 | 134/1350 [01:53<14:53, 1.36it/s] |
|
2024-02-27 15:50:24,061 - INFO: train loss: 0.45: 15%|#4 | 201/1350 [02:28<14:11, 1.35it/s] |
|
2024-02-27 15:50:38,964 - INFO: train loss: 0.45: 15%|#4 | 201/1350 [02:43<14:11, 1.35it/s] |
|
2024-02-27 15:51:15,137 - INFO: train loss: 0.35: 20%|#9 | 268/1350 [03:19<13:30, 1.33it/s] |
|
2024-02-27 15:51:28,966 - INFO: train loss: 0.35: 20%|#9 | 268/1350 [03:33<13:30, 1.33it/s] |
|
2024-02-27 15:52:04,065 - INFO: train loss: 0.29: 25%|##4 | 335/1350 [04:08<12:33, 1.35it/s] |
|
2024-02-27 15:52:18,967 - INFO: train loss: 0.29: 25%|##4 | 335/1350 [04:23<12:33, 1.35it/s] |
|
2024-02-27 15:52:54,392 - INFO: train loss: 0.27: 30%|##9 | 402/1350 [04:58<11:46, 1.34it/s] |
|
2024-02-27 15:53:08,968 - INFO: train loss: 0.27: 30%|##9 | 402/1350 [05:13<11:46, 1.34it/s] |
|
2024-02-27 15:53:42,207 - INFO: train loss: 0.39: 35%|###4 | 469/1350 [05:46<10:47, 1.36it/s] |
|
2024-02-27 15:53:58,970 - INFO: train loss: 0.39: 35%|###4 | 469/1350 [06:03<10:47, 1.36it/s] |
|
2024-02-27 15:54:31,674 - INFO: train loss: 0.41: 40%|###9 | 536/1350 [06:36<09:59, 1.36it/s] |
|
2024-02-27 15:54:48,971 - INFO: train loss: 0.41: 40%|###9 | 536/1350 [06:53<09:59, 1.36it/s] |
|
2024-02-27 15:55:19,071 - INFO: train loss: 0.30: 45%|####4 | 603/1350 [07:23<09:03, 1.38it/s] |
|
2024-02-27 15:55:29,613 - INFO: train loss: 0.30: 45%|####4 | 603/1350 [07:33<09:03, 1.38it/s] |
|
2024-02-27 15:56:08,097 - INFO: train loss: 0.38: 50%|####9 | 670/1350 [08:12<08:15, 1.37it/s] |
|
2024-02-27 15:56:18,973 - INFO: train loss: 0.38: 50%|####9 | 670/1350 [08:23<08:15, 1.37it/s] |
|
2024-02-27 15:56:57,480 - INFO: train loss: 0.30: 55%|#####4 | 737/1350 [09:01<07:28, 1.37it/s] |
|
2024-02-27 15:57:08,974 - INFO: train loss: 0.30: 55%|#####4 | 737/1350 [09:13<07:28, 1.37it/s] |
|
2024-02-27 15:57:46,055 - INFO: train loss: 0.40: 60%|#####9 | 804/1350 [09:50<06:38, 1.37it/s] |
|
2024-02-27 15:57:58,976 - INFO: train loss: 0.40: 60%|#####9 | 804/1350 [10:03<06:38, 1.37it/s] |
|
2024-02-27 15:58:35,524 - INFO: train loss: 0.27: 65%|######4 | 871/1350 [10:39<05:50, 1.37it/s] |
|
2024-02-27 15:58:48,977 - INFO: train loss: 0.27: 65%|######4 | 871/1350 [10:53<05:50, 1.37it/s] |
|
2024-02-27 15:59:24,087 - INFO: train loss: 0.41: 69%|######9 | 938/1350 [11:28<05:00, 1.37it/s] |
|
2024-02-27 15:59:38,978 - INFO: train loss: 0.41: 69%|######9 | 938/1350 [11:43<05:00, 1.37it/s] |
|
2024-02-27 16:00:13,900 - INFO: train loss: 0.54: 74%|#######4 | 1005/1350 [12:18<04:13, 1.36it/s] |
|
2024-02-27 16:00:28,980 - INFO: train loss: 0.54: 74%|#######4 | 1005/1350 [12:33<04:13, 1.36it/s] |
|
2024-02-27 16:01:02,605 - INFO: train loss: 0.31: 79%|#######9 | 1072/1350 [13:06<03:23, 1.37it/s] |
|
2024-02-27 16:01:18,981 - INFO: train loss: 0.31: 79%|#######9 | 1072/1350 [13:23<03:23, 1.37it/s] |
|
2024-02-27 16:01:52,569 - INFO: train loss: 0.44: 84%|########4 | 1139/1350 [13:56<02:35, 1.36it/s] |
|
2024-02-27 16:02:08,982 - INFO: train loss: 0.44: 84%|########4 | 1139/1350 [14:13<02:35, 1.36it/s] |
|
2024-02-27 16:02:40,755 - INFO: train loss: 0.17: 89%|########9 | 1206/1350 [14:45<01:45, 1.37it/s] |
|
2024-02-27 16:02:50,929 - INFO: train loss: 0.17: 89%|########9 | 1206/1350 [14:55<01:45, 1.37it/s] |
|
2024-02-27 16:03:28,505 - INFO: train loss: 0.27: 94%|#########4| 1273/1350 [15:32<00:55, 1.38it/s] |
|
2024-02-27 16:03:38,984 - INFO: train loss: 0.27: 94%|#########4| 1273/1350 [15:43<00:55, 1.38it/s] |
|
2024-02-27 16:04:17,776 - INFO: train loss: 0.32: 99%|#########9| 1340/1350 [16:22<00:07, 1.37it/s] |
|
2024-02-27 16:04:24,908 - INFO: train loss: 0.25: 100%|##########| 1350/1350 [16:29<00:00, 1.36it/s] |
|
2024-02-27 16:04:24,912 - INFO: Starting validation inference |
|
2024-02-27 16:04:24,912 - INFO: validation progress: 0%| | 0/1200 [00:00<?, ?it/s] |
|
2024-02-27 16:04:29,109 - INFO: validation progress: 5%|5 | 60/1200 [00:04<01:19, 14.30it/s] |
|
2024-02-27 16:04:33,272 - INFO: validation progress: 10%|# | 120/1200 [00:08<01:15, 14.36it/s] |
|
2024-02-27 16:04:37,356 - INFO: validation progress: 15%|#5 | 180/1200 [00:12<01:10, 14.51it/s] |
|
2024-02-27 16:04:41,494 - INFO: validation progress: 20%|## | 240/1200 [00:16<01:06, 14.51it/s] |
|
2024-02-27 16:04:45,651 - INFO: validation progress: 25%|##5 | 300/1200 [00:20<01:02, 14.48it/s] |
|
2024-02-27 16:04:49,770 - INFO: validation progress: 30%|### | 360/1200 [00:24<00:57, 14.51it/s] |
|
2024-02-27 16:04:53,936 - INFO: validation progress: 35%|###5 | 420/1200 [00:29<00:53, 14.47it/s] |
|
2024-02-27 16:04:58,040 - INFO: validation progress: 40%|#### | 480/1200 [00:33<00:49, 14.52it/s] |
|
2024-02-27 16:05:02,194 - INFO: validation progress: 45%|####5 | 540/1200 [00:37<00:45, 14.50it/s] |
|
2024-02-27 16:05:06,361 - INFO: validation progress: 50%|##### | 600/1200 [00:41<00:41, 14.47it/s] |
|
2024-02-27 16:05:10,516 - INFO: validation progress: 55%|#####5 | 660/1200 [00:45<00:37, 14.46it/s] |
|
2024-02-27 16:05:14,669 - INFO: validation progress: 60%|###### | 720/1200 [00:49<00:33, 14.46it/s] |
|
2024-02-27 16:05:18,765 - INFO: validation progress: 65%|######5 | 780/1200 [00:53<00:28, 14.51it/s] |
|
2024-02-27 16:05:22,909 - INFO: validation progress: 70%|####### | 840/1200 [00:57<00:24, 14.50it/s] |
|
2024-02-27 16:05:27,077 - INFO: validation progress: 75%|#######5 | 900/1200 [01:02<00:20, 14.47it/s] |
|
2024-02-27 16:05:31,204 - INFO: validation progress: 80%|######## | 960/1200 [01:06<00:16, 14.49it/s] |
|
2024-02-27 16:05:35,351 - INFO: validation progress: 85%|########5 | 1020/1200 [01:10<00:12, 14.48it/s] |
|
2024-02-27 16:05:39,503 - INFO: validation progress: 90%|######### | 1080/1200 [01:14<00:08, 14.47it/s] |
|
2024-02-27 16:05:43,621 - INFO: validation progress: 95%|#########5| 1140/1200 [01:18<00:04, 14.50it/s] |
|
2024-02-27 16:05:47,698 - INFO: validation progress: 100%|##########| 1200/1200 [01:22<00:00, 14.57it/s] |
|
2024-02-27 16:05:47,700 - INFO: validation progress: 100%|##########| 1200/1200 [01:22<00:00, 14.49it/s] |
|
2024-02-27 16:05:47,727 - INFO: Mean validation loss: 0.31232 |
|
2024-02-27 16:05:47,729 - INFO: Validation AUC: 0.97734 |
|
2024-02-27 16:05:47,767 - INFO: Saving best model checkpoint: val_AUC 0.97231 -> 0.97734 to /app/output |
|
2024-02-27 16:06:01,990 - INFO: Training Epoch: 4 / 15 |
|
2024-02-27 16:06:01,990 - INFO: train loss: 0%| | 0/1350 [00:00<?, ?it/s] |
|
2024-02-27 16:06:02,016 - INFO: Evaluation step: 1350 |
|
2024-02-27 16:06:51,334 - INFO: train loss: 0.21: 5%|4 | 67/1350 [00:49<15:44, 1.36it/s] |
|
2024-02-27 16:07:09,168 - INFO: train loss: 0.21: 5%|4 | 67/1350 [01:07<15:44, 1.36it/s] |
|
2024-02-27 16:07:40,541 - INFO: train loss: 0.09: 10%|9 | 134/1350 [01:38<14:54, 1.36it/s] |
|
2024-02-27 16:07:50,950 - INFO: train loss: 0.09: 10%|9 | 134/1350 [01:48<14:54, 1.36it/s] |
|
2024-02-27 16:08:30,081 - INFO: train loss: 0.19: 15%|#4 | 201/1350 [02:28<14:06, 1.36it/s] |
|
2024-02-27 16:08:40,951 - INFO: train loss: 0.19: 15%|#4 | 201/1350 [02:38<14:06, 1.36it/s] |
|
2024-02-27 16:09:18,496 - INFO: train loss: 0.24: 20%|#9 | 268/1350 [03:16<13:11, 1.37it/s] |
|
2024-02-27 16:09:29,171 - INFO: train loss: 0.24: 20%|#9 | 268/1350 [03:27<13:11, 1.37it/s] |
|
2024-02-27 16:10:08,931 - INFO: train loss: 0.14: 25%|##4 | 335/1350 [04:06<12:30, 1.35it/s] |
|
2024-02-27 16:10:19,173 - INFO: train loss: 0.14: 25%|##4 | 335/1350 [04:17<12:30, 1.35it/s] |
|
2024-02-27 16:10:57,913 - INFO: train loss: 0.22: 30%|##9 | 402/1350 [04:55<11:38, 1.36it/s] |
|
2024-02-27 16:11:09,174 - INFO: train loss: 0.22: 30%|##9 | 402/1350 [05:07<11:38, 1.36it/s] |
|
2024-02-27 16:11:46,772 - INFO: train loss: 0.20: 35%|###4 | 469/1350 [05:44<10:46, 1.36it/s] |
|
2024-02-27 16:11:59,175 - INFO: train loss: 0.20: 35%|###4 | 469/1350 [05:57<10:46, 1.36it/s] |
|
2024-02-27 16:12:35,227 - INFO: train loss: 0.14: 40%|###9 | 536/1350 [06:33<09:54, 1.37it/s] |
|
2024-02-27 16:12:49,176 - INFO: train loss: 0.14: 40%|###9 | 536/1350 [06:47<09:54, 1.37it/s] |
|
2024-02-27 16:13:24,332 - INFO: train loss: 0.12: 45%|####4 | 603/1350 [07:22<09:06, 1.37it/s] |
|
2024-02-27 16:13:39,178 - INFO: train loss: 0.12: 45%|####4 | 603/1350 [07:37<09:06, 1.37it/s] |
|
2024-02-27 16:14:12,476 - INFO: train loss: 0.17: 50%|####9 | 670/1350 [08:10<08:14, 1.37it/s] |
|
2024-02-27 16:14:29,179 - INFO: train loss: 0.17: 50%|####9 | 670/1350 [08:27<08:14, 1.37it/s] |
|
2024-02-27 16:15:01,820 - INFO: train loss: 0.18: 55%|#####4 | 737/1350 [08:59<07:27, 1.37it/s] |
|
2024-02-27 16:15:19,180 - INFO: train loss: 0.18: 55%|#####4 | 737/1350 [09:17<07:27, 1.37it/s] |
|
2024-02-27 16:15:52,248 - INFO: train loss: 0.14: 60%|#####9 | 804/1350 [09:50<06:42, 1.36it/s] |
|
2024-02-27 16:16:09,182 - INFO: train loss: 0.14: 60%|#####9 | 804/1350 [10:07<06:42, 1.36it/s] |
|
2024-02-27 16:16:40,738 - INFO: train loss: 0.21: 65%|######4 | 871/1350 [10:38<05:51, 1.36it/s] |
|
2024-02-27 16:16:50,962 - INFO: train loss: 0.21: 65%|######4 | 871/1350 [10:48<05:51, 1.36it/s] |
|
2024-02-27 16:17:31,430 - INFO: train loss: 0.28: 69%|######9 | 938/1350 [11:29<05:04, 1.35it/s] |
|
2024-02-27 16:17:49,184 - INFO: train loss: 0.28: 69%|######9 | 938/1350 [11:47<05:04, 1.35it/s] |
|
2024-02-27 16:18:19,938 - INFO: train loss: 0.27: 74%|#######4 | 1005/1350 [12:17<04:13, 1.36it/s] |
|
2024-02-27 16:18:30,964 - INFO: train loss: 0.27: 74%|#######4 | 1005/1350 [12:28<04:13, 1.36it/s] |
|
2024-02-27 16:19:07,292 - INFO: train loss: 0.27: 79%|#######9 | 1072/1350 [13:05<03:22, 1.38it/s] |
|
2024-02-27 16:19:19,186 - INFO: train loss: 0.27: 79%|#######9 | 1072/1350 [13:17<03:22, 1.38it/s] |
|
2024-02-27 16:19:57,865 - INFO: train loss: 0.37: 84%|########4 | 1139/1350 [13:55<02:35, 1.36it/s] |
|
2024-02-27 16:20:09,188 - INFO: train loss: 0.37: 84%|########4 | 1139/1350 [14:07<02:35, 1.36it/s] |
|
2024-02-27 16:20:47,020 - INFO: train loss: 0.27: 89%|########9 | 1206/1350 [14:45<01:45, 1.36it/s] |
|
2024-02-27 16:20:59,189 - INFO: train loss: 0.27: 89%|########9 | 1206/1350 [14:57<01:45, 1.36it/s] |
|
2024-02-27 16:21:35,850 - INFO: train loss: 0.16: 94%|#########4| 1273/1350 [15:33<00:56, 1.36it/s] |
|
2024-02-27 16:21:49,190 - INFO: train loss: 0.16: 94%|#########4| 1273/1350 [15:47<00:56, 1.36it/s] |
|
2024-02-27 16:22:25,717 - INFO: train loss: 0.25: 99%|#########9| 1340/1350 [16:23<00:07, 1.36it/s] |
|
2024-02-27 16:22:33,016 - INFO: train loss: 0.23: 100%|##########| 1350/1350 [16:31<00:00, 1.36it/s] |
|
2024-02-27 16:22:33,020 - INFO: Starting validation inference |
|
2024-02-27 16:22:33,021 - INFO: validation progress: 0%| | 0/1200 [00:00<?, ?it/s] |
|
2024-02-27 16:22:37,246 - INFO: validation progress: 5%|5 | 60/1200 [00:04<01:20, 14.20it/s] |
|
2024-02-27 16:22:41,458 - INFO: validation progress: 10%|# | 120/1200 [00:08<01:15, 14.23it/s] |
|
2024-02-27 16:22:45,584 - INFO: validation progress: 15%|#5 | 180/1200 [00:12<01:10, 14.37it/s] |
|
2024-02-27 16:22:49,733 - INFO: validation progress: 20%|## | 240/1200 [00:16<01:06, 14.41it/s] |
|
2024-02-27 16:22:53,907 - INFO: validation progress: 25%|##5 | 300/1200 [00:20<01:02, 14.39it/s] |
|
2024-02-27 16:22:58,054 - INFO: validation progress: 30%|### | 360/1200 [00:25<00:58, 14.42it/s] |
|
2024-02-27 16:23:02,243 - INFO: validation progress: 35%|###5 | 420/1200 [00:29<00:54, 14.39it/s] |
|
2024-02-27 16:23:06,387 - INFO: validation progress: 40%|#### | 480/1200 [00:33<00:49, 14.42it/s] |
|
2024-02-27 16:23:10,552 - INFO: validation progress: 45%|####5 | 540/1200 [00:37<00:45, 14.41it/s] |
|
2024-02-27 16:23:14,718 - INFO: validation progress: 50%|##### | 600/1200 [00:41<00:41, 14.41it/s] |
|
2024-02-27 16:23:18,878 - INFO: validation progress: 55%|#####5 | 660/1200 [00:45<00:37, 14.41it/s] |
|
2024-02-27 16:23:23,033 - INFO: validation progress: 60%|###### | 720/1200 [00:50<00:33, 14.42it/s] |
|
2024-02-27 16:23:27,102 - INFO: validation progress: 65%|######5 | 780/1200 [00:54<00:28, 14.52it/s] |
|
2024-02-27 16:23:31,237 - INFO: validation progress: 70%|####### | 840/1200 [00:58<00:24, 14.52it/s] |
|
2024-02-27 16:23:35,394 - INFO: validation progress: 75%|#######5 | 900/1200 [01:02<00:20, 14.49it/s] |
|
2024-02-27 16:23:39,511 - INFO: validation progress: 80%|######## | 960/1200 [01:06<00:16, 14.52it/s] |
|
2024-02-27 16:23:43,677 - INFO: validation progress: 85%|########5 | 1020/1200 [01:10<00:12, 14.48it/s] |
|
2024-02-27 16:23:47,838 - INFO: validation progress: 90%|######### | 1080/1200 [01:14<00:08, 14.46it/s] |
|
2024-02-27 16:23:52,005 - INFO: validation progress: 95%|#########5| 1140/1200 [01:18<00:04, 14.44it/s] |
|
2024-02-27 16:23:56,078 - INFO: validation progress: 100%|##########| 1200/1200 [01:23<00:00, 14.53it/s] |
|
2024-02-27 16:23:56,081 - INFO: validation progress: 100%|##########| 1200/1200 [01:23<00:00, 14.45it/s] |
|
2024-02-27 16:23:56,107 - INFO: Mean validation loss: 0.32958 |
|
2024-02-27 16:23:56,109 - INFO: Validation AUC: 0.97797 |
|
2024-02-27 16:23:56,147 - INFO: Saving best model checkpoint: val_AUC 0.97734 -> 0.97797 to /app/output |
|
2024-02-27 16:24:10,405 - INFO: Training Epoch: 5 / 15 |
|
2024-02-27 16:24:10,405 - INFO: train loss: 0%| | 0/1350 [00:00<?, ?it/s] |
|
2024-02-27 16:24:10,432 - INFO: Evaluation step: 1350 |
|
2024-02-27 16:24:59,544 - INFO: train loss: 0.10: 5%|4 | 67/1350 [00:49<15:40, 1.36it/s] |
|
2024-02-27 16:25:09,742 - INFO: train loss: 0.10: 5%|4 | 67/1350 [00:59<15:40, 1.36it/s] |
|
2024-02-27 16:25:48,261 - INFO: train loss: 0.14: 10%|9 | 134/1350 [01:37<14:47, 1.37it/s] |
|
2024-02-27 16:25:59,335 - INFO: train loss: 0.14: 10%|9 | 134/1350 [01:48<14:47, 1.37it/s] |
|
2024-02-27 16:26:36,364 - INFO: train loss: 0.05: 15%|#4 | 201/1350 [02:25<13:52, 1.38it/s] |
|
2024-02-27 16:26:49,337 - INFO: train loss: 0.05: 15%|#4 | 201/1350 [02:38<13:52, 1.38it/s] |
|
2024-02-27 16:27:24,644 - INFO: train loss: 0.15: 20%|#9 | 268/1350 [03:14<13:02, 1.38it/s] |
|
2024-02-27 16:27:39,338 - INFO: train loss: 0.15: 20%|#9 | 268/1350 [03:28<13:02, 1.38it/s] |
|
2024-02-27 16:28:13,602 - INFO: train loss: 0.17: 25%|##4 | 335/1350 [04:03<12:16, 1.38it/s] |
|
2024-02-27 16:28:29,339 - INFO: train loss: 0.17: 25%|##4 | 335/1350 [04:18<12:16, 1.38it/s] |
|
2024-02-27 16:29:02,658 - INFO: train loss: 0.13: 30%|##9 | 402/1350 [04:52<11:30, 1.37it/s] |
|
2024-02-27 16:29:19,341 - INFO: train loss: 0.13: 30%|##9 | 402/1350 [05:08<11:30, 1.37it/s] |
|
2024-02-27 16:29:50,139 - INFO: train loss: 0.12: 35%|###4 | 469/1350 [05:39<10:35, 1.39it/s] |
|
2024-02-27 16:30:01,076 - INFO: train loss: 0.12: 35%|###4 | 469/1350 [05:50<10:35, 1.39it/s] |
|
2024-02-27 16:30:39,323 - INFO: train loss: 0.10: 40%|###9 | 536/1350 [06:28<09:50, 1.38it/s] |
|
2024-02-27 16:30:49,343 - INFO: train loss: 0.10: 40%|###9 | 536/1350 [06:38<09:50, 1.38it/s] |
|
2024-02-27 16:31:27,602 - INFO: train loss: 0.31: 45%|####4 | 603/1350 [07:17<09:00, 1.38it/s] |
|
2024-02-27 16:31:39,344 - INFO: train loss: 0.31: 45%|####4 | 603/1350 [07:28<09:00, 1.38it/s] |
|
2024-02-27 16:32:17,637 - INFO: train loss: 0.16: 50%|####9 | 670/1350 [08:07<08:17, 1.37it/s] |
|
2024-02-27 16:32:29,345 - INFO: train loss: 0.16: 50%|####9 | 670/1350 [08:18<08:17, 1.37it/s] |
|
2024-02-27 16:33:06,454 - INFO: train loss: 0.14: 55%|#####4 | 737/1350 [08:56<07:27, 1.37it/s] |
|
2024-02-27 16:33:19,346 - INFO: train loss: 0.14: 55%|#####4 | 737/1350 [09:08<07:27, 1.37it/s] |
|
2024-02-27 16:33:56,015 - INFO: train loss: 0.15: 60%|#####9 | 804/1350 [09:45<06:40, 1.36it/s] |
|
2024-02-27 16:34:09,347 - INFO: train loss: 0.15: 60%|#####9 | 804/1350 [09:58<06:40, 1.36it/s] |
|
2024-02-27 16:34:44,988 - INFO: train loss: 0.07: 65%|######4 | 871/1350 [10:34<05:50, 1.37it/s] |
|
2024-02-27 16:34:59,349 - INFO: train loss: 0.07: 65%|######4 | 871/1350 [10:48<05:50, 1.37it/s] |
|
2024-02-27 16:35:32,609 - INFO: train loss: 0.20: 69%|######9 | 938/1350 [11:22<04:59, 1.38it/s] |
|
2024-02-27 16:35:49,350 - INFO: train loss: 0.20: 69%|######9 | 938/1350 [11:38<04:59, 1.38it/s] |
|
2024-02-27 16:36:22,715 - INFO: train loss: 0.20: 74%|#######4 | 1005/1350 [12:12<04:12, 1.37it/s] |
|
2024-02-27 16:36:39,351 - INFO: train loss: 0.20: 74%|#######4 | 1005/1350 [12:28<04:12, 1.37it/s] |
|
2024-02-27 16:37:11,405 - INFO: train loss: 0.15: 79%|#######9 | 1072/1350 [13:00<03:23, 1.37it/s] |
|
2024-02-27 16:37:29,352 - INFO: train loss: 0.15: 79%|#######9 | 1072/1350 [13:18<03:23, 1.37it/s] |
|
2024-02-27 16:38:01,672 - INFO: train loss: 0.18: 84%|########4 | 1139/1350 [13:51<02:35, 1.36it/s] |
|
2024-02-27 16:38:19,354 - INFO: train loss: 0.18: 84%|########4 | 1139/1350 [14:08<02:35, 1.36it/s] |
|
2024-02-27 16:38:51,447 - INFO: train loss: 0.11: 89%|########9 | 1206/1350 [14:41<01:46, 1.35it/s] |
|
2024-02-27 16:39:09,355 - INFO: train loss: 0.11: 89%|########9 | 1206/1350 [14:58<01:46, 1.35it/s] |
|
2024-02-27 16:39:39,914 - INFO: train loss: 0.27: 94%|#########4| 1273/1350 [15:29<00:56, 1.36it/s] |
|
2024-02-27 16:39:51,088 - INFO: train loss: 0.27: 94%|#########4| 1273/1350 [15:40<00:56, 1.36it/s] |
|
2024-02-27 16:40:29,266 - INFO: train loss: 0.19: 99%|#########9| 1340/1350 [16:18<00:07, 1.36it/s] |
|
2024-02-27 16:40:36,395 - INFO: train loss: 0.12: 100%|##########| 1350/1350 [16:25<00:00, 1.37it/s] |
|
2024-02-27 16:40:36,399 - INFO: Starting validation inference |
|
2024-02-27 16:40:36,400 - INFO: validation progress: 0%| | 0/1200 [00:00<?, ?it/s] |
|
2024-02-27 16:40:40,614 - INFO: validation progress: 5%|5 | 60/1200 [00:04<01:20, 14.24it/s] |
|
2024-02-27 16:40:44,788 - INFO: validation progress: 10%|# | 120/1200 [00:08<01:15, 14.32it/s] |
|
2024-02-27 16:40:48,902 - INFO: validation progress: 15%|#5 | 180/1200 [00:12<01:10, 14.44it/s] |
|
2024-02-27 16:40:53,062 - INFO: validation progress: 20%|## | 240/1200 [00:16<01:06, 14.43it/s] |
|
2024-02-27 16:40:57,260 - INFO: validation progress: 25%|##5 | 300/1200 [00:20<01:02, 14.38it/s] |
|
2024-02-27 16:41:01,451 - INFO: validation progress: 30%|### | 360/1200 [00:25<00:58, 14.36it/s] |
|
2024-02-27 16:41:05,664 - INFO: validation progress: 35%|###5 | 420/1200 [00:29<00:54, 14.32it/s] |
|
2024-02-27 16:41:09,828 - INFO: validation progress: 40%|#### | 480/1200 [00:33<00:50, 14.35it/s] |
|
2024-02-27 16:41:14,059 - INFO: validation progress: 45%|####5 | 540/1200 [00:37<00:46, 14.30it/s] |
|
2024-02-27 16:41:18,271 - INFO: validation progress: 50%|##### | 600/1200 [00:41<00:42, 14.28it/s] |
|
2024-02-27 16:41:22,464 - INFO: validation progress: 55%|#####5 | 660/1200 [00:46<00:37, 14.29it/s] |
|
2024-02-27 16:41:26,652 - INFO: validation progress: 60%|###### | 720/1200 [00:50<00:33, 14.30it/s] |
|
2024-02-27 16:41:30,779 - INFO: validation progress: 65%|######5 | 780/1200 [00:54<00:29, 14.37it/s] |
|
2024-02-27 16:41:34,974 - INFO: validation progress: 70%|####### | 840/1200 [00:58<00:25, 14.35it/s] |
|
2024-02-27 16:41:39,160 - INFO: validation progress: 75%|#######5 | 900/1200 [01:02<00:20, 14.35it/s] |
|
2024-02-27 16:41:43,322 - INFO: validation progress: 80%|######## | 960/1200 [01:06<00:16, 14.37it/s] |
|
2024-02-27 16:41:47,499 - INFO: validation progress: 85%|########5 | 1020/1200 [01:11<00:12, 14.37it/s] |
|
2024-02-27 16:41:51,688 - INFO: validation progress: 90%|######### | 1080/1200 [01:15<00:08, 14.35it/s] |
|
2024-02-27 16:41:55,833 - INFO: validation progress: 95%|#########5| 1140/1200 [01:19<00:04, 14.39it/s] |
|
2024-02-27 16:41:59,938 - INFO: validation progress: 100%|##########| 1200/1200 [01:23<00:00, 14.46it/s] |
|
2024-02-27 16:41:59,941 - INFO: validation progress: 100%|##########| 1200/1200 [01:23<00:00, 14.36it/s] |
|
2024-02-27 16:41:59,968 - INFO: Mean validation loss: 0.42099 |
|
2024-02-27 16:41:59,970 - INFO: Validation AUC: 0.97725 |
|
2024-02-27 16:42:00,034 - INFO: Training Epoch: 6 / 15 |
|
2024-02-27 16:42:00,034 - INFO: train loss: 0%| | 0/1350 [00:00<?, ?it/s] |
|
2024-02-27 16:42:00,061 - INFO: Evaluation step: 1350 |
|
2024-02-27 16:42:49,227 - INFO: train loss: 0.08: 5%|4 | 67/1350 [00:49<15:41, 1.36it/s] |
|
2024-02-27 16:42:59,360 - INFO: train loss: 0.08: 5%|4 | 67/1350 [00:59<15:41, 1.36it/s] |
|
2024-02-27 16:43:37,853 - INFO: train loss: 0.07: 10%|9 | 134/1350 [01:37<14:46, 1.37it/s] |
|
2024-02-27 16:43:49,361 - INFO: train loss: 0.07: 10%|9 | 134/1350 [01:49<14:46, 1.37it/s] |
|
2024-02-27 16:44:28,092 - INFO: train loss: 0.07: 15%|#4 | 201/1350 [02:28<14:08, 1.35it/s] |
|
2024-02-27 16:44:39,362 - INFO: train loss: 0.07: 15%|#4 | 201/1350 [02:39<14:08, 1.35it/s] |
|
2024-02-27 16:45:19,117 - INFO: train loss: 0.07: 20%|#9 | 268/1350 [03:19<13:29, 1.34it/s] |
|
2024-02-27 16:45:29,363 - INFO: train loss: 0.07: 20%|#9 | 268/1350 [03:29<13:29, 1.34it/s] |
|
2024-02-27 16:46:08,582 - INFO: train loss: 0.10: 25%|##4 | 335/1350 [04:08<12:35, 1.34it/s] |
|
2024-02-27 16:46:19,364 - INFO: train loss: 0.10: 25%|##4 | 335/1350 [04:19<12:35, 1.34it/s] |
|
2024-02-27 16:46:56,973 - INFO: train loss: 0.03: 30%|##9 | 402/1350 [04:56<11:38, 1.36it/s] |
|
2024-02-27 16:47:09,366 - INFO: train loss: 0.03: 30%|##9 | 402/1350 [05:09<11:38, 1.36it/s] |
|
2024-02-27 16:47:45,197 - INFO: train loss: 0.05: 35%|###4 | 469/1350 [05:45<10:44, 1.37it/s] |
|
2024-02-27 16:47:59,367 - INFO: train loss: 0.05: 35%|###4 | 469/1350 [05:59<10:44, 1.37it/s] |
|
2024-02-27 16:48:32,037 - INFO: train loss: 0.04: 40%|###9 | 536/1350 [06:32<09:46, 1.39it/s] |
|
2024-02-27 16:48:49,368 - INFO: train loss: 0.04: 40%|###9 | 536/1350 [06:49<09:46, 1.39it/s] |
|
2024-02-27 16:49:22,511 - INFO: train loss: 0.05: 45%|####4 | 603/1350 [07:22<09:06, 1.37it/s] |
|
2024-02-27 16:49:39,369 - INFO: train loss: 0.05: 45%|####4 | 603/1350 [07:39<09:06, 1.37it/s] |
|
2024-02-27 16:50:11,618 - INFO: train loss: 0.06: 50%|####9 | 670/1350 [08:11<08:17, 1.37it/s] |
|
2024-02-27 16:50:29,370 - INFO: train loss: 0.06: 50%|####9 | 670/1350 [08:29<08:17, 1.37it/s] |
|
2024-02-27 16:51:00,061 - INFO: train loss: 0.04: 55%|#####4 | 737/1350 [09:00<07:26, 1.37it/s] |
|
2024-02-27 16:51:11,102 - INFO: train loss: 0.04: 55%|#####4 | 737/1350 [09:11<07:26, 1.37it/s] |
|
2024-02-27 16:51:50,850 - INFO: train loss: 0.12: 60%|#####9 | 804/1350 [09:50<06:42, 1.36it/s] |
|
2024-02-27 16:52:01,104 - INFO: train loss: 0.12: 60%|#####9 | 804/1350 [10:01<06:42, 1.36it/s] |
|
2024-02-27 16:52:40,355 - INFO: train loss: 0.04: 65%|######4 | 871/1350 [10:40<05:53, 1.35it/s] |
|
2024-02-27 16:52:51,105 - INFO: train loss: 0.04: 65%|######4 | 871/1350 [10:51<05:53, 1.35it/s] |
|
2024-02-27 16:53:28,731 - INFO: train loss: 0.11: 69%|######9 | 938/1350 [11:28<05:02, 1.36it/s] |
|
2024-02-27 16:53:39,374 - INFO: train loss: 0.11: 69%|######9 | 938/1350 [11:39<05:02, 1.36it/s] |
|
2024-02-27 16:54:18,008 - INFO: train loss: 0.09: 74%|#######4 | 1005/1350 [12:17<04:13, 1.36it/s] |
|
2024-02-27 16:54:29,376 - INFO: train loss: 0.09: 74%|#######4 | 1005/1350 [12:29<04:13, 1.36it/s] |
|
2024-02-27 16:55:06,585 - INFO: train loss: 0.09: 79%|#######9 | 1072/1350 [13:06<03:23, 1.37it/s] |
|
2024-02-27 16:55:19,377 - INFO: train loss: 0.09: 79%|#######9 | 1072/1350 [13:19<03:23, 1.37it/s] |
|
2024-02-27 16:55:56,984 - INFO: train loss: 0.04: 84%|########4 | 1139/1350 [13:56<02:35, 1.36it/s] |
|
2024-02-27 16:56:09,378 - INFO: train loss: 0.04: 84%|########4 | 1139/1350 [14:09<02:35, 1.36it/s] |
|
2024-02-27 16:56:45,946 - INFO: train loss: 0.11: 89%|########9 | 1206/1350 [14:45<01:45, 1.36it/s] |
|
2024-02-27 16:56:59,380 - INFO: train loss: 0.11: 89%|########9 | 1206/1350 [14:59<01:45, 1.36it/s] |
|
2024-02-27 16:57:35,033 - INFO: train loss: 0.14: 94%|#########4| 1273/1350 [15:34<00:56, 1.36it/s] |
|
2024-02-27 16:57:49,381 - INFO: train loss: 0.14: 94%|#########4| 1273/1350 [15:49<00:56, 1.36it/s] |
|
2024-02-27 16:58:24,675 - INFO: train loss: 0.12: 99%|#########9| 1340/1350 [16:24<00:07, 1.36it/s] |
|
2024-02-27 16:58:31,829 - INFO: train loss: 0.11: 100%|##########| 1350/1350 [16:31<00:00, 1.36it/s] |
|
2024-02-27 16:58:31,834 - INFO: Starting validation inference |
|
2024-02-27 16:58:31,834 - INFO: validation progress: 0%| | 0/1200 [00:00<?, ?it/s] |
|
2024-02-27 16:58:36,109 - INFO: validation progress: 5%|5 | 60/1200 [00:04<01:21, 14.03it/s] |
|
2024-02-27 16:58:40,332 - INFO: validation progress: 10%|# | 120/1200 [00:08<01:16, 14.14it/s] |
|
2024-02-27 16:58:44,481 - INFO: validation progress: 15%|#5 | 180/1200 [00:12<01:11, 14.28it/s] |
|
2024-02-27 16:58:48,657 - INFO: validation progress: 20%|## | 240/1200 [00:16<01:07, 14.32it/s] |
|
2024-02-27 16:58:52,875 - INFO: validation progress: 25%|##5 | 300/1200 [00:21<01:03, 14.28it/s] |
|
2024-02-27 16:58:57,054 - INFO: validation progress: 30%|### | 360/1200 [00:25<00:58, 14.31it/s] |
|
2024-02-27 16:59:01,279 - INFO: validation progress: 35%|###5 | 420/1200 [00:29<00:54, 14.27it/s] |
|
2024-02-27 16:59:05,453 - INFO: validation progress: 40%|#### | 480/1200 [00:33<00:50, 14.31it/s] |
|
2024-02-27 16:59:09,657 - INFO: validation progress: 45%|####5 | 540/1200 [00:37<00:46, 14.29it/s] |
|
2024-02-27 16:59:13,870 - INFO: validation progress: 50%|##### | 600/1200 [00:42<00:42, 14.28it/s] |
|
2024-02-27 16:59:18,072 - INFO: validation progress: 55%|#####5 | 660/1200 [00:46<00:37, 14.28it/s] |
|
2024-02-27 16:59:22,276 - INFO: validation progress: 60%|###### | 720/1200 [00:50<00:33, 14.28it/s] |
|
2024-02-27 16:59:26,439 - INFO: validation progress: 65%|######5 | 780/1200 [00:54<00:29, 14.32it/s] |
|
2024-02-27 16:59:30,653 - INFO: validation progress: 70%|####### | 840/1200 [00:58<00:25, 14.29it/s] |
|
2024-02-27 16:59:34,874 - INFO: validation progress: 75%|#######5 | 900/1200 [01:03<00:21, 14.27it/s] |
|
2024-02-27 16:59:39,056 - INFO: validation progress: 80%|######## | 960/1200 [01:07<00:16, 14.29it/s] |
|
2024-02-27 16:59:43,268 - INFO: validation progress: 85%|########5 | 1020/1200 [01:11<00:12, 14.28it/s] |
|
2024-02-27 16:59:47,480 - INFO: validation progress: 90%|######### | 1080/1200 [01:15<00:08, 14.27it/s] |
|
2024-02-27 16:59:51,675 - INFO: validation progress: 95%|#########5| 1140/1200 [01:19<00:04, 14.28it/s] |
|
2024-02-27 16:59:55,799 - INFO: validation progress: 100%|##########| 1200/1200 [01:23<00:00, 14.36it/s] |
|
2024-02-27 16:59:55,802 - INFO: validation progress: 100%|##########| 1200/1200 [01:23<00:00, 14.29it/s] |
|
2024-02-27 16:59:55,829 - INFO: Mean validation loss: 0.35996 |
|
2024-02-27 16:59:55,831 - INFO: Validation AUC: 0.98081 |
|
2024-02-27 16:59:55,869 - INFO: Saving best model checkpoint: val_AUC 0.97797 -> 0.98081 to /app/output |
|
2024-02-27 17:00:10,151 - INFO: Training Epoch: 7 / 15 |
|
2024-02-27 17:00:10,151 - INFO: train loss: 0%| | 0/1350 [00:00<?, ?it/s] |
|
2024-02-27 17:00:10,178 - INFO: Evaluation step: 1350 |
|
2024-02-27 17:00:59,169 - INFO: train loss: 0.04: 5%|4 | 67/1350 [00:49<15:38, 1.37it/s] |
|
2024-02-27 17:01:09,481 - INFO: train loss: 0.04: 5%|4 | 67/1350 [00:59<15:38, 1.37it/s] |
|
2024-02-27 17:01:48,570 - INFO: train loss: 0.07: 10%|9 | 134/1350 [01:38<14:53, 1.36it/s] |
|
2024-02-27 17:01:59,483 - INFO: train loss: 0.07: 10%|9 | 134/1350 [01:49<14:53, 1.36it/s] |
|
2024-02-27 17:02:37,008 - INFO: train loss: 0.07: 15%|#4 | 201/1350 [02:26<13:58, 1.37it/s] |
|
2024-02-27 17:02:49,484 - INFO: train loss: 0.07: 15%|#4 | 201/1350 [02:39<13:58, 1.37it/s] |
|
2024-02-27 17:03:26,520 - INFO: train loss: 0.01: 20%|#9 | 268/1350 [03:16<13:13, 1.36it/s] |
|
2024-02-27 17:03:39,485 - INFO: train loss: 0.01: 20%|#9 | 268/1350 [03:29<13:13, 1.36it/s] |
|
2024-02-27 17:04:15,557 - INFO: train loss: 0.03: 25%|##4 | 335/1350 [04:05<12:23, 1.36it/s] |
|
2024-02-27 17:04:29,487 - INFO: train loss: 0.03: 25%|##4 | 335/1350 [04:19<12:23, 1.36it/s] |
|
2024-02-27 17:05:03,749 - INFO: train loss: 0.02: 30%|##9 | 402/1350 [04:53<11:30, 1.37it/s] |
|
2024-02-27 17:05:19,488 - INFO: train loss: 0.02: 30%|##9 | 402/1350 [05:09<11:30, 1.37it/s] |
|
2024-02-27 17:05:52,427 - INFO: train loss: 0.10: 35%|###4 | 469/1350 [05:42<10:41, 1.37it/s] |
|
2024-02-27 17:06:09,489 - INFO: train loss: 0.10: 35%|###4 | 469/1350 [05:59<10:41, 1.37it/s] |
|
2024-02-27 17:06:40,530 - INFO: train loss: 0.21: 40%|###9 | 536/1350 [06:30<09:49, 1.38it/s] |
|
2024-02-27 17:06:51,200 - INFO: train loss: 0.21: 40%|###9 | 536/1350 [06:41<09:49, 1.38it/s] |
|
2024-02-27 17:07:29,889 - INFO: train loss: 0.03: 45%|####4 | 603/1350 [07:19<09:04, 1.37it/s] |
|
2024-02-27 17:07:39,895 - INFO: train loss: 0.03: 45%|####4 | 603/1350 [07:29<09:04, 1.37it/s] |
|
2024-02-27 17:08:19,218 - INFO: train loss: 0.03: 50%|####9 | 670/1350 [08:09<08:16, 1.37it/s] |
|
2024-02-27 17:08:29,492 - INFO: train loss: 0.03: 50%|####9 | 670/1350 [08:19<08:16, 1.37it/s] |
|
2024-02-27 17:09:09,175 - INFO: train loss: 0.12: 55%|#####4 | 737/1350 [08:59<07:30, 1.36it/s] |
|
2024-02-27 17:09:19,493 - INFO: train loss: 0.12: 55%|#####4 | 737/1350 [09:09<07:30, 1.36it/s] |
|
2024-02-27 17:09:58,540 - INFO: train loss: 0.04: 60%|#####9 | 804/1350 [09:48<06:41, 1.36it/s] |
|
2024-02-27 17:10:09,495 - INFO: train loss: 0.04: 60%|#####9 | 804/1350 [09:59<06:41, 1.36it/s] |
|
2024-02-27 17:10:46,803 - INFO: train loss: 0.11: 65%|######4 | 871/1350 [10:36<05:50, 1.37it/s] |
|
2024-02-27 17:10:59,496 - INFO: train loss: 0.11: 65%|######4 | 871/1350 [10:49<05:50, 1.37it/s] |
|
2024-02-27 17:11:35,507 - INFO: train loss: 0.01: 69%|######9 | 938/1350 [11:25<05:00, 1.37it/s] |
|
2024-02-27 17:11:49,497 - INFO: train loss: 0.01: 69%|######9 | 938/1350 [11:39<05:00, 1.37it/s] |
|
2024-02-27 17:12:25,717 - INFO: train loss: 0.03: 74%|#######4 | 1005/1350 [12:15<04:13, 1.36it/s] |
|
2024-02-27 17:12:39,498 - INFO: train loss: 0.03: 74%|#######4 | 1005/1350 [12:29<04:13, 1.36it/s] |
|
2024-02-27 17:13:14,554 - INFO: train loss: 0.05: 79%|#######9 | 1072/1350 [13:04<03:23, 1.36it/s] |
|
2024-02-27 17:13:29,499 - INFO: train loss: 0.05: 79%|#######9 | 1072/1350 [13:19<03:23, 1.36it/s] |
|
2024-02-27 17:14:03,633 - INFO: train loss: 0.12: 84%|########4 | 1139/1350 [13:53<02:34, 1.36it/s] |
|
2024-02-27 17:14:19,500 - INFO: train loss: 0.12: 84%|########4 | 1139/1350 [14:09<02:34, 1.36it/s] |
|
2024-02-27 17:14:53,167 - INFO: train loss: 0.10: 89%|########9 | 1206/1350 [14:43<01:45, 1.36it/s] |
|
2024-02-27 17:15:09,502 - INFO: train loss: 0.10: 89%|########9 | 1206/1350 [14:59<01:45, 1.36it/s] |
|
2024-02-27 17:15:42,796 - INFO: train loss: 0.05: 94%|#########4| 1273/1350 [15:32<00:56, 1.36it/s] |
|
2024-02-27 17:15:59,503 - INFO: train loss: 0.05: 94%|#########4| 1273/1350 [15:49<00:56, 1.36it/s] |
|
2024-02-27 17:16:32,697 - INFO: train loss: 0.14: 99%|#########9| 1340/1350 [16:22<00:07, 1.35it/s] |
|
2024-02-27 17:16:40,047 - INFO: train loss: 0.06: 100%|##########| 1350/1350 [16:29<00:00, 1.36it/s] |
|
2024-02-27 17:16:40,051 - INFO: Starting validation inference |
|
2024-02-27 17:16:40,051 - INFO: validation progress: 0%| | 0/1200 [00:00<?, ?it/s] |
|
2024-02-27 17:16:44,321 - INFO: validation progress: 5%|5 | 60/1200 [00:04<01:21, 14.05it/s] |
|
2024-02-27 17:16:48,551 - INFO: validation progress: 10%|# | 120/1200 [00:08<01:16, 14.13it/s] |
|
2024-02-27 17:16:52,701 - INFO: validation progress: 15%|#5 | 180/1200 [00:12<01:11, 14.28it/s] |
|
2024-02-27 17:16:56,882 - INFO: validation progress: 20%|## | 240/1200 [00:16<01:07, 14.31it/s] |
|
2024-02-27 17:17:01,089 - INFO: validation progress: 25%|##5 | 300/1200 [00:21<01:02, 14.29it/s] |
|
2024-02-27 17:17:05,291 - INFO: validation progress: 30%|### | 360/1200 [00:25<00:58, 14.29it/s] |
|
2024-02-27 17:17:09,499 - INFO: validation progress: 35%|###5 | 420/1200 [00:29<00:54, 14.28it/s] |
|
2024-02-27 17:17:13,685 - INFO: validation progress: 40%|#### | 480/1200 [00:33<00:50, 14.30it/s] |
|
2024-02-27 17:17:17,876 - INFO: validation progress: 45%|####5 | 540/1200 [00:37<00:46, 14.30it/s] |
|
2024-02-27 17:17:22,089 - INFO: validation progress: 50%|##### | 600/1200 [00:42<00:42, 14.28it/s] |
|
2024-02-27 17:17:26,301 - INFO: validation progress: 55%|#####5 | 660/1200 [00:46<00:37, 14.27it/s] |
|
2024-02-27 17:17:30,509 - INFO: validation progress: 60%|###### | 720/1200 [00:50<00:33, 14.27it/s] |
|
2024-02-27 17:17:34,664 - INFO: validation progress: 65%|######5 | 780/1200 [00:54<00:29, 14.32it/s] |
|
2024-02-27 17:17:38,873 - INFO: validation progress: 70%|####### | 840/1200 [00:58<00:25, 14.30it/s] |
|
2024-02-27 17:17:43,093 - INFO: validation progress: 75%|#######5 | 900/1200 [01:03<00:21, 14.27it/s] |
|
2024-02-27 17:17:47,273 - INFO: validation progress: 80%|######## | 960/1200 [01:07<00:16, 14.30it/s] |
|
2024-02-27 17:17:51,495 - INFO: validation progress: 85%|########5 | 1020/1200 [01:11<00:12, 14.27it/s] |
|
2024-02-27 17:17:55,696 - INFO: validation progress: 90%|######### | 1080/1200 [01:15<00:08, 14.28it/s] |
|
2024-02-27 17:17:59,877 - INFO: validation progress: 95%|#########5| 1140/1200 [01:19<00:04, 14.30it/s] |
|
2024-02-27 17:18:04,011 - INFO: validation progress: 100%|##########| 1200/1200 [01:23<00:00, 14.36it/s] |
|
2024-02-27 17:18:04,015 - INFO: validation progress: 100%|##########| 1200/1200 [01:23<00:00, 14.29it/s] |
|
2024-02-27 17:18:04,042 - INFO: Mean validation loss: 0.52982 |
|
2024-02-27 17:18:04,044 - INFO: Validation AUC: 0.97502 |
|
2024-02-27 17:18:04,249 - INFO: Training Epoch: 8 / 15 |
|
2024-02-27 17:18:04,249 - INFO: train loss: 0%| | 0/1350 [00:00<?, ?it/s] |
|
2024-02-27 17:18:04,277 - INFO: Evaluation step: 1350 |
|
2024-02-27 17:18:54,074 - INFO: train loss: 0.05: 5%|4 | 67/1350 [00:49<15:54, 1.34it/s] |
|
2024-02-27 17:19:09,507 - INFO: train loss: 0.05: 5%|4 | 67/1350 [01:05<15:54, 1.34it/s] |
|
2024-02-27 17:19:43,628 - INFO: train loss: 0.01: 10%|9 | 134/1350 [01:39<15:01, 1.35it/s] |
|
2024-02-27 17:19:59,508 - INFO: train loss: 0.01: 10%|9 | 134/1350 [01:55<15:01, 1.35it/s] |
|
2024-02-27 17:20:32,566 - INFO: train loss: 0.03: 15%|#4 | 201/1350 [02:28<14:06, 1.36it/s] |
|
2024-02-27 17:20:49,509 - INFO: train loss: 0.03: 15%|#4 | 201/1350 [02:45<14:06, 1.36it/s] |
|
2024-02-27 17:21:23,691 - INFO: train loss: 0.13: 20%|#9 | 268/1350 [03:19<13:28, 1.34it/s] |
|
2024-02-27 17:21:39,511 - INFO: train loss: 0.13: 20%|#9 | 268/1350 [03:35<13:28, 1.34it/s] |
|
2024-02-27 17:22:13,406 - INFO: train loss: 0.08: 25%|##4 | 335/1350 [04:09<12:36, 1.34it/s] |
|
2024-02-27 17:22:29,512 - INFO: train loss: 0.08: 25%|##4 | 335/1350 [04:25<12:36, 1.34it/s] |
|
2024-02-27 17:23:01,748 - INFO: train loss: 0.04: 30%|##9 | 402/1350 [04:57<11:38, 1.36it/s] |
|
2024-02-27 17:23:19,514 - INFO: train loss: 0.04: 30%|##9 | 402/1350 [05:15<11:38, 1.36it/s] |
|
2024-02-27 17:23:51,493 - INFO: train loss: 0.02: 35%|###4 | 469/1350 [05:47<10:50, 1.35it/s] |
|
2024-02-27 17:24:09,515 - INFO: train loss: 0.02: 35%|###4 | 469/1350 [06:05<10:50, 1.35it/s] |
|
2024-02-27 17:24:39,130 - INFO: train loss: 0.05: 40%|###9 | 536/1350 [06:34<09:54, 1.37it/s] |
|
2024-02-27 17:24:49,516 - INFO: train loss: 0.05: 40%|###9 | 536/1350 [06:45<09:54, 1.37it/s] |
|
2024-02-27 17:25:28,234 - INFO: train loss: 0.11: 45%|####4 | 603/1350 [07:23<09:05, 1.37it/s] |
|
2024-02-27 17:25:39,517 - INFO: train loss: 0.11: 45%|####4 | 603/1350 [07:35<09:05, 1.37it/s] |
|
2024-02-27 17:26:18,704 - INFO: train loss: 0.03: 50%|####9 | 670/1350 [08:14<08:21, 1.36it/s] |
|
2024-02-27 17:26:29,518 - INFO: train loss: 0.03: 50%|####9 | 670/1350 [08:25<08:21, 1.36it/s] |
|
2024-02-27 17:27:08,194 - INFO: train loss: 0.03: 55%|#####4 | 737/1350 [09:03<07:32, 1.35it/s] |
|
2024-02-27 17:27:19,520 - INFO: train loss: 0.03: 55%|#####4 | 737/1350 [09:15<07:32, 1.35it/s] |
|
2024-02-27 17:27:56,263 - INFO: train loss: 0.07: 60%|#####9 | 804/1350 [09:52<06:39, 1.37it/s] |
|
2024-02-27 17:28:09,521 - INFO: train loss: 0.07: 60%|#####9 | 804/1350 [10:05<06:39, 1.37it/s] |
|
2024-02-27 17:28:47,023 - INFO: train loss: 0.02: 65%|######4 | 871/1350 [10:42<05:54, 1.35it/s] |
|
2024-02-27 17:28:59,522 - INFO: train loss: 0.02: 65%|######4 | 871/1350 [10:55<05:54, 1.35it/s] |
|
2024-02-27 17:29:37,288 - INFO: train loss: 0.03: 69%|######9 | 938/1350 [11:33<05:06, 1.35it/s] |
|
2024-02-27 17:29:49,524 - INFO: train loss: 0.03: 69%|######9 | 938/1350 [11:45<05:06, 1.35it/s] |
|
2024-02-27 17:30:25,864 - INFO: train loss: 0.01: 74%|#######4 | 1005/1350 [12:21<04:14, 1.36it/s] |
|
2024-02-27 17:30:39,525 - INFO: train loss: 0.01: 74%|#######4 | 1005/1350 [12:35<04:14, 1.36it/s] |
|
2024-02-27 17:31:13,197 - INFO: train loss: 0.12: 79%|#######9 | 1072/1350 [13:08<03:22, 1.37it/s] |
|
2024-02-27 17:31:29,526 - INFO: train loss: 0.12: 79%|#######9 | 1072/1350 [13:25<03:22, 1.37it/s] |
|
2024-02-27 17:32:03,012 - INFO: train loss: 0.08: 84%|########4 | 1139/1350 [13:58<02:34, 1.36it/s] |
|
2024-02-27 17:32:19,528 - INFO: train loss: 0.08: 84%|########4 | 1139/1350 [14:15<02:34, 1.36it/s] |
|
2024-02-27 17:32:52,710 - INFO: train loss: 0.07: 89%|########9 | 1206/1350 [14:48<01:45, 1.36it/s] |
|
2024-02-27 17:33:09,531 - INFO: train loss: 0.07: 89%|########9 | 1206/1350 [15:05<01:45, 1.36it/s] |
|
2024-02-27 17:33:41,708 - INFO: train loss: 0.08: 94%|#########4| 1273/1350 [15:37<00:56, 1.36it/s] |
|
2024-02-27 17:33:59,532 - INFO: train loss: 0.08: 94%|#########4| 1273/1350 [15:55<00:56, 1.36it/s] |
|
2024-02-27 17:34:30,578 - INFO: train loss: 0.10: 99%|#########9| 1340/1350 [16:26<00:07, 1.36it/s] |
|
2024-02-27 17:34:37,264 - INFO: train loss: 0.20: 100%|##########| 1350/1350 [16:33<00:00, 1.36it/s] |
|
2024-02-27 17:34:37,268 - INFO: Starting validation inference |
|
2024-02-27 17:34:37,269 - INFO: validation progress: 0%| | 0/1200 [00:00<?, ?it/s] |
|
2024-02-27 17:34:41,588 - INFO: validation progress: 5%|5 | 60/1200 [00:04<01:22, 13.89it/s] |
|
2024-02-27 17:34:45,889 - INFO: validation progress: 10%|# | 120/1200 [00:08<01:17, 13.93it/s] |
|
2024-02-27 17:34:50,070 - INFO: validation progress: 15%|#5 | 180/1200 [00:12<01:12, 14.12it/s] |
|
2024-02-27 17:34:54,292 - INFO: validation progress: 20%|## | 240/1200 [00:17<01:07, 14.15it/s] |
|
2024-02-27 17:34:58,557 - INFO: validation progress: 25%|##5 | 300/1200 [00:21<01:03, 14.12it/s] |
|
2024-02-27 17:35:02,799 - INFO: validation progress: 30%|### | 360/1200 [00:25<00:59, 14.13it/s] |
|
2024-02-27 17:35:07,075 - INFO: validation progress: 35%|###5 | 420/1200 [00:29<00:55, 14.10it/s] |
|
2024-02-27 17:35:11,305 - INFO: validation progress: 40%|#### | 480/1200 [00:34<00:50, 14.13it/s] |
|
2024-02-27 17:35:15,556 - INFO: validation progress: 45%|####5 | 540/1200 [00:38<00:46, 14.12it/s] |
|
2024-02-27 17:35:19,820 - INFO: validation progress: 50%|##### | 600/1200 [00:42<00:42, 14.11it/s] |
|
2024-02-27 17:35:24,117 - INFO: validation progress: 55%|#####5 | 660/1200 [00:46<00:38, 14.06it/s] |
|
2024-02-27 17:35:28,369 - INFO: validation progress: 60%|###### | 720/1200 [00:51<00:34, 14.08it/s] |
|
2024-02-27 17:35:32,566 - INFO: validation progress: 65%|######5 | 780/1200 [00:55<00:29, 14.14it/s] |
|
2024-02-27 17:35:36,825 - INFO: validation progress: 70%|####### | 840/1200 [00:59<00:25, 14.13it/s] |
|
2024-02-27 17:35:41,096 - INFO: validation progress: 75%|#######5 | 900/1200 [01:03<00:21, 14.10it/s] |
|
2024-02-27 17:35:45,324 - INFO: validation progress: 80%|######## | 960/1200 [01:08<00:16, 14.13it/s] |
|
2024-02-27 17:35:49,577 - INFO: validation progress: 85%|########5 | 1020/1200 [01:12<00:12, 14.12it/s] |
|
2024-02-27 17:35:53,830 - INFO: validation progress: 90%|######### | 1080/1200 [01:16<00:08, 14.12it/s] |
|
2024-02-27 17:35:58,078 - INFO: validation progress: 95%|#########5| 1140/1200 [01:20<00:04, 14.12it/s] |
|
2024-02-27 17:36:02,289 - INFO: validation progress: 100%|##########| 1200/1200 [01:25<00:00, 14.16it/s] |
|
2024-02-27 17:36:02,293 - INFO: validation progress: 100%|##########| 1200/1200 [01:25<00:00, 14.11it/s] |
|
2024-02-27 17:36:02,321 - INFO: Mean validation loss: 0.46195 |
|
2024-02-27 17:36:02,323 - INFO: Validation AUC: 0.98454 |
|
2024-02-27 17:36:02,361 - INFO: Saving best model checkpoint: val_AUC 0.98081 -> 0.98454 to /app/output |
|
2024-02-27 17:36:16,570 - INFO: Training Epoch: 9 / 15 |
|
2024-02-27 17:36:16,571 - INFO: train loss: 0%| | 0/1350 [00:00<?, ?it/s] |
|
2024-02-27 17:36:16,597 - INFO: Evaluation step: 1350 |
|
2024-02-27 17:37:06,880 - INFO: train loss: 0.10: 5%|4 | 67/1350 [00:50<16:03, 1.33it/s] |
|
2024-02-27 17:37:19,544 - INFO: train loss: 0.10: 5%|4 | 67/1350 [01:02<16:03, 1.33it/s] |
|
2024-02-27 17:37:55,697 - INFO: train loss: 0.04: 10%|9 | 134/1350 [01:39<14:57, 1.36it/s] |
|
2024-02-27 17:38:09,546 - INFO: train loss: 0.04: 10%|9 | 134/1350 [01:52<14:57, 1.36it/s] |
|
2024-02-27 17:38:45,033 - INFO: train loss: 0.01: 15%|#4 | 201/1350 [02:28<14:06, 1.36it/s] |
|
2024-02-27 17:38:59,547 - INFO: train loss: 0.01: 15%|#4 | 201/1350 [02:42<14:06, 1.36it/s] |
|
2024-02-27 17:39:35,036 - INFO: train loss: 0.00: 20%|#9 | 268/1350 [03:18<13:21, 1.35it/s] |
|
2024-02-27 17:39:49,548 - INFO: train loss: 0.00: 20%|#9 | 268/1350 [03:32<13:21, 1.35it/s] |
|
2024-02-27 17:40:24,263 - INFO: train loss: 0.02: 25%|##4 | 335/1350 [04:07<12:29, 1.35it/s] |
|
2024-02-27 17:40:39,549 - INFO: train loss: 0.02: 25%|##4 | 335/1350 [04:22<12:29, 1.35it/s] |
|
2024-02-27 17:41:13,948 - INFO: train loss: 0.08: 30%|##9 | 402/1350 [04:57<11:41, 1.35it/s] |
|
2024-02-27 17:41:29,551 - INFO: train loss: 0.08: 30%|##9 | 402/1350 [05:12<11:41, 1.35it/s] |
|
2024-02-27 17:42:02,899 - INFO: train loss: 0.05: 35%|###4 | 469/1350 [05:46<10:48, 1.36it/s] |
|
2024-02-27 17:42:19,552 - INFO: train loss: 0.05: 35%|###4 | 469/1350 [06:02<10:48, 1.36it/s] |
|
2024-02-27 17:42:51,035 - INFO: train loss: 0.06: 40%|###9 | 536/1350 [06:34<09:54, 1.37it/s] |
|
2024-02-27 17:43:01,296 - INFO: train loss: 0.06: 40%|###9 | 536/1350 [06:44<09:54, 1.37it/s] |
|
2024-02-27 17:43:40,623 - INFO: train loss: 0.03: 45%|####4 | 603/1350 [07:24<09:08, 1.36it/s] |
|
2024-02-27 17:43:51,298 - INFO: train loss: 0.03: 45%|####4 | 603/1350 [07:34<09:08, 1.36it/s] |
|
2024-02-27 17:44:29,100 - INFO: train loss: 0.03: 50%|####9 | 670/1350 [08:12<08:16, 1.37it/s] |
|
2024-02-27 17:44:39,555 - INFO: train loss: 0.03: 50%|####9 | 670/1350 [08:22<08:16, 1.37it/s] |
|
2024-02-27 17:45:18,795 - INFO: train loss: 0.01: 55%|#####4 | 737/1350 [09:02<07:29, 1.36it/s] |
|
2024-02-27 17:45:29,557 - INFO: train loss: 0.01: 55%|#####4 | 737/1350 [09:12<07:29, 1.36it/s] |
|
2024-02-27 17:46:09,755 - INFO: train loss: 0.03: 60%|#####9 | 804/1350 [09:53<06:45, 1.35it/s] |
|
2024-02-27 17:46:19,953 - INFO: train loss: 0.03: 60%|#####9 | 804/1350 [10:03<06:45, 1.35it/s] |
|
2024-02-27 17:46:59,813 - INFO: train loss: 0.01: 65%|######4 | 871/1350 [10:43<05:56, 1.34it/s] |
|
2024-02-27 17:47:09,955 - INFO: train loss: 0.01: 65%|######4 | 871/1350 [10:53<05:56, 1.34it/s] |
|
2024-02-27 17:47:49,508 - INFO: train loss: 0.06: 69%|######9 | 938/1350 [11:32<05:06, 1.35it/s] |
|
2024-02-27 17:47:59,561 - INFO: train loss: 0.06: 69%|######9 | 938/1350 [11:42<05:06, 1.35it/s] |
|
2024-02-27 17:48:38,156 - INFO: train loss: 0.03: 74%|#######4 | 1005/1350 [12:21<04:14, 1.36it/s] |
|
2024-02-27 17:48:49,562 - INFO: train loss: 0.03: 74%|#######4 | 1005/1350 [12:32<04:14, 1.36it/s] |
|
2024-02-27 17:49:27,757 - INFO: train loss: 0.07: 79%|#######9 | 1072/1350 [13:11<03:25, 1.35it/s] |
|
2024-02-27 17:49:39,563 - INFO: train loss: 0.07: 79%|#######9 | 1072/1350 [13:22<03:25, 1.35it/s] |
|
2024-02-27 17:50:17,103 - INFO: train loss: 0.02: 84%|########4 | 1139/1350 [14:00<02:35, 1.36it/s] |
|
2024-02-27 17:50:29,565 - INFO: train loss: 0.02: 84%|########4 | 1139/1350 [14:12<02:35, 1.36it/s] |
|
2024-02-27 17:51:03,868 - INFO: train loss: 0.10: 89%|########9 | 1206/1350 [14:47<01:44, 1.38it/s] |
|
2024-02-27 17:51:19,566 - INFO: train loss: 0.10: 89%|########9 | 1206/1350 [15:02<01:44, 1.38it/s] |
|
2024-02-27 17:51:53,738 - INFO: train loss: 0.07: 94%|#########4| 1273/1350 [15:37<00:56, 1.37it/s] |
|
2024-02-27 17:52:09,568 - INFO: train loss: 0.07: 94%|#########4| 1273/1350 [15:52<00:56, 1.37it/s] |
|
2024-02-27 17:52:42,157 - INFO: train loss: 0.08: 99%|#########9| 1340/1350 [16:25<00:07, 1.37it/s] |
|
2024-02-27 17:52:49,906 - INFO: train loss: 0.04: 100%|##########| 1350/1350 [16:33<00:00, 1.36it/s] |
|
2024-02-27 17:52:49,911 - INFO: Starting validation inference |
|
2024-02-27 17:52:49,911 - INFO: validation progress: 0%| | 0/1200 [00:00<?, ?it/s] |
|
2024-02-27 17:52:54,209 - INFO: validation progress: 5%|5 | 60/1200 [00:04<01:21, 13.96it/s] |
|
2024-02-27 17:52:58,475 - INFO: validation progress: 10%|# | 120/1200 [00:08<01:17, 14.02it/s] |
|
2024-02-27 17:53:02,676 - INFO: validation progress: 15%|#5 | 180/1200 [00:12<01:12, 14.14it/s] |
|
2024-02-27 17:53:06,916 - INFO: validation progress: 20%|## | 240/1200 [00:17<01:07, 14.14it/s] |
|
2024-02-27 17:53:11,176 - INFO: validation progress: 25%|##5 | 300/1200 [00:21<01:03, 14.12it/s] |
|
2024-02-27 17:53:15,399 - INFO: validation progress: 30%|### | 360/1200 [00:25<00:59, 14.15it/s] |
|
2024-02-27 17:53:19,642 - INFO: validation progress: 35%|###5 | 420/1200 [00:29<00:55, 14.15it/s] |
|
2024-02-27 17:53:23,864 - INFO: validation progress: 40%|#### | 480/1200 [00:33<00:50, 14.17it/s] |
|
2024-02-27 17:53:28,136 - INFO: validation progress: 45%|####5 | 540/1200 [00:38<00:46, 14.13it/s] |
|
2024-02-27 17:53:32,398 - INFO: validation progress: 50%|##### | 600/1200 [00:42<00:42, 14.11it/s] |
|
2024-02-27 17:53:36,669 - INFO: validation progress: 55%|#####5 | 660/1200 [00:46<00:38, 14.09it/s] |
|
2024-02-27 17:53:40,923 - INFO: validation progress: 60%|###### | 720/1200 [00:51<00:34, 14.10it/s] |
|
2024-02-27 17:53:45,101 - INFO: validation progress: 65%|######5 | 780/1200 [00:55<00:29, 14.18it/s] |
|
2024-02-27 17:53:49,334 - INFO: validation progress: 70%|####### | 840/1200 [00:59<00:25, 14.17it/s] |
|
2024-02-27 17:53:53,603 - INFO: validation progress: 75%|#######5 | 900/1200 [01:03<00:21, 14.14it/s] |
|
2024-02-27 17:53:57,853 - INFO: validation progress: 80%|######## | 960/1200 [01:07<00:16, 14.13it/s] |
|
2024-02-27 17:54:02,109 - INFO: validation progress: 85%|########5 | 1020/1200 [01:12<00:12, 14.12it/s] |
|
2024-02-27 17:54:06,373 - INFO: validation progress: 90%|######### | 1080/1200 [01:16<00:08, 14.11it/s] |
|
2024-02-27 17:54:10,624 - INFO: validation progress: 95%|#########5| 1140/1200 [01:20<00:04, 14.11it/s] |
|
2024-02-27 17:54:14,806 - INFO: validation progress: 100%|##########| 1200/1200 [01:24<00:00, 14.18it/s] |
|
2024-02-27 17:54:14,811 - INFO: validation progress: 100%|##########| 1200/1200 [01:24<00:00, 14.13it/s] |
|
2024-02-27 17:54:14,839 - INFO: Mean validation loss: 0.55605 |
|
2024-02-27 17:54:14,841 - INFO: Validation AUC: 0.98378 |
|
2024-02-27 17:54:14,908 - INFO: Training Epoch: 10 / 15 |
|
2024-02-27 17:54:14,908 - INFO: train loss: 0%| | 0/1350 [00:00<?, ?it/s] |
|
2024-02-27 17:54:14,937 - INFO: Evaluation step: 1350 |
|
2024-02-27 17:55:04,346 - INFO: train loss: 0.03: 5%|4 | 67/1350 [00:49<15:46, 1.36it/s] |
|
2024-02-27 17:55:19,574 - INFO: train loss: 0.03: 5%|4 | 67/1350 [01:04<15:46, 1.36it/s] |
|
2024-02-27 17:55:53,948 - INFO: train loss: 0.09: 10%|9 | 134/1350 [01:39<14:59, 1.35it/s] |
|
2024-02-27 17:56:09,576 - INFO: train loss: 0.09: 10%|9 | 134/1350 [01:54<14:59, 1.35it/s] |
|
2024-02-27 17:56:43,757 - INFO: train loss: 0.06: 15%|#4 | 201/1350 [02:28<14:11, 1.35it/s] |
|
2024-02-27 17:56:59,577 - INFO: train loss: 0.06: 15%|#4 | 201/1350 [02:44<14:11, 1.35it/s] |
|
2024-02-27 17:57:33,782 - INFO: train loss: 0.06: 20%|#9 | 268/1350 [03:18<13:24, 1.35it/s] |
|
2024-02-27 17:57:49,578 - INFO: train loss: 0.06: 20%|#9 | 268/1350 [03:34<13:24, 1.35it/s] |
|
2024-02-27 17:58:22,901 - INFO: train loss: 0.01: 25%|##4 | 335/1350 [04:07<12:30, 1.35it/s] |
|
2024-02-27 17:58:39,579 - INFO: train loss: 0.01: 25%|##4 | 335/1350 [04:24<12:30, 1.35it/s] |
|
2024-02-27 17:59:13,756 - INFO: train loss: 0.08: 30%|##9 | 402/1350 [04:58<11:47, 1.34it/s] |
|
2024-02-27 17:59:29,580 - INFO: train loss: 0.08: 30%|##9 | 402/1350 [05:14<11:47, 1.34it/s] |
|
2024-02-27 18:00:03,506 - INFO: train loss: 0.04: 35%|###4 | 469/1350 [05:48<10:56, 1.34it/s] |
|
2024-02-27 18:00:19,582 - INFO: train loss: 0.04: 35%|###4 | 469/1350 [06:04<10:56, 1.34it/s] |
|
2024-02-27 18:00:52,786 - INFO: train loss: 0.13: 40%|###9 | 536/1350 [06:37<10:03, 1.35it/s] |
|
2024-02-27 18:01:09,583 - INFO: train loss: 0.13: 40%|###9 | 536/1350 [06:54<10:03, 1.35it/s] |
|
2024-02-27 18:01:41,699 - INFO: train loss: 0.07: 45%|####4 | 603/1350 [07:26<09:11, 1.35it/s] |
|
2024-02-27 18:01:59,584 - INFO: train loss: 0.07: 45%|####4 | 603/1350 [07:44<09:11, 1.35it/s] |
|
2024-02-27 18:02:31,773 - INFO: train loss: 0.20: 50%|####9 | 670/1350 [08:16<08:23, 1.35it/s] |
|
2024-02-27 18:02:49,585 - INFO: train loss: 0.20: 50%|####9 | 670/1350 [08:34<08:23, 1.35it/s] |
|
2024-02-27 18:03:21,265 - INFO: train loss: 0.14: 55%|#####4 | 737/1350 [09:06<07:33, 1.35it/s] |
|
2024-02-27 18:03:31,324 - INFO: train loss: 0.14: 55%|#####4 | 737/1350 [09:16<07:33, 1.35it/s] |
|
2024-02-27 18:04:11,276 - INFO: train loss: 0.09: 60%|#####9 | 804/1350 [09:56<06:45, 1.35it/s] |
|
2024-02-27 18:04:21,326 - INFO: train loss: 0.09: 60%|#####9 | 804/1350 [10:06<06:45, 1.35it/s] |
|
2024-02-27 18:05:02,239 - INFO: train loss: 0.04: 65%|######4 | 871/1350 [10:47<05:58, 1.34it/s] |
|
2024-02-27 18:05:19,588 - INFO: train loss: 0.04: 65%|######4 | 871/1350 [11:04<05:58, 1.34it/s] |
|
2024-02-27 18:05:51,313 - INFO: train loss: 0.01: 69%|######9 | 938/1350 [11:36<05:06, 1.35it/s] |
|
2024-02-27 18:06:01,328 - INFO: train loss: 0.01: 69%|######9 | 938/1350 [11:46<05:06, 1.35it/s] |
|
2024-02-27 18:06:39,474 - INFO: train loss: 0.00: 74%|#######4 | 1005/1350 [12:24<04:13, 1.36it/s] |
|
2024-02-27 18:06:49,590 - INFO: train loss: 0.00: 74%|#######4 | 1005/1350 [12:34<04:13, 1.36it/s] |
|
2024-02-27 18:07:29,181 - INFO: train loss: 0.01: 79%|#######9 | 1072/1350 [13:14<03:25, 1.36it/s] |
|
2024-02-27 18:07:39,591 - INFO: train loss: 0.01: 79%|#######9 | 1072/1350 [13:24<03:25, 1.36it/s] |
|
2024-02-27 18:08:17,728 - INFO: train loss: 0.01: 84%|########4 | 1139/1350 [14:02<02:34, 1.36it/s] |
|
2024-02-27 18:08:29,592 - INFO: train loss: 0.01: 84%|########4 | 1139/1350 [14:14<02:34, 1.36it/s] |
|
2024-02-27 18:09:06,737 - INFO: train loss: 0.05: 89%|########9 | 1206/1350 [14:51<01:45, 1.36it/s] |
|
2024-02-27 18:09:19,594 - INFO: train loss: 0.05: 89%|########9 | 1206/1350 [15:04<01:45, 1.36it/s] |
|
2024-02-27 18:09:54,878 - INFO: train loss: 0.01: 94%|#########4| 1273/1350 [15:39<00:56, 1.37it/s] |
|
2024-02-27 18:10:09,595 - INFO: train loss: 0.01: 94%|#########4| 1273/1350 [15:54<00:56, 1.37it/s] |
|
2024-02-27 18:10:44,764 - INFO: train loss: 0.07: 99%|#########9| 1340/1350 [16:29<00:07, 1.36it/s] |
|
2024-02-27 18:10:51,812 - INFO: train loss: 0.17: 100%|##########| 1350/1350 [16:36<00:00, 1.35it/s] |
|
2024-02-27 18:10:51,817 - INFO: Starting validation inference |
|
2024-02-27 18:10:51,817 - INFO: validation progress: 0%| | 0/1200 [00:00<?, ?it/s] |
|
2024-02-27 18:10:56,173 - INFO: validation progress: 5%|5 | 60/1200 [00:04<01:22, 13.78it/s] |
|
2024-02-27 18:11:00,504 - INFO: validation progress: 10%|# | 120/1200 [00:08<01:18, 13.82it/s] |
|
2024-02-27 18:11:04,750 - INFO: validation progress: 15%|#5 | 180/1200 [00:12<01:13, 13.96it/s] |
|
2024-02-27 18:11:09,020 - INFO: validation progress: 20%|## | 240/1200 [00:17<01:08, 14.00it/s] |
|
2024-02-27 18:11:13,373 - INFO: validation progress: 25%|##5 | 300/1200 [00:21<01:04, 13.92it/s] |
|
2024-02-27 18:11:17,664 - INFO: validation progress: 30%|### | 360/1200 [00:25<01:00, 13.94it/s] |
|
2024-02-27 18:11:21,982 - INFO: validation progress: 35%|###5 | 420/1200 [00:30<00:56, 13.93it/s] |
|
2024-02-27 18:11:26,261 - INFO: validation progress: 40%|#### | 480/1200 [00:34<00:51, 13.96it/s] |
|
2024-02-27 18:11:30,556 - INFO: validation progress: 45%|####5 | 540/1200 [00:38<00:47, 13.96it/s] |
|
2024-02-27 18:11:34,873 - INFO: validation progress: 50%|##### | 600/1200 [00:43<00:43, 13.94it/s] |
|
2024-02-27 18:11:39,182 - INFO: validation progress: 55%|#####5 | 660/1200 [00:47<00:38, 13.94it/s] |
|
2024-02-27 18:11:43,490 - INFO: validation progress: 60%|###### | 720/1200 [00:51<00:34, 13.93it/s] |
|
2024-02-27 18:11:47,738 - INFO: validation progress: 65%|######5 | 780/1200 [00:55<00:30, 13.99it/s] |
|
2024-02-27 18:11:52,070 - INFO: validation progress: 70%|####### | 840/1200 [01:00<00:25, 13.95it/s] |
|
2024-02-27 18:11:56,406 - INFO: validation progress: 75%|#######5 | 900/1200 [01:04<00:21, 13.91it/s] |
|
2024-02-27 18:12:00,707 - INFO: validation progress: 80%|######## | 960/1200 [01:08<00:17, 13.93it/s] |
|
2024-02-27 18:12:05,023 - INFO: validation progress: 85%|########5 | 1020/1200 [01:13<00:12, 13.92it/s] |
|
2024-02-27 18:12:09,336 - INFO: validation progress: 90%|######### | 1080/1200 [01:17<00:08, 13.92it/s] |
|
2024-02-27 18:12:13,653 - INFO: validation progress: 95%|#########5| 1140/1200 [01:21<00:04, 13.91it/s] |
|
2024-02-27 18:12:17,888 - INFO: validation progress: 100%|##########| 1200/1200 [01:26<00:00, 13.99it/s] |
|
2024-02-27 18:12:17,892 - INFO: validation progress: 100%|##########| 1200/1200 [01:26<00:00, 13.94it/s] |
|
2024-02-27 18:12:17,921 - INFO: Mean validation loss: 0.53012 |
|
2024-02-27 18:12:17,923 - INFO: Validation AUC: 0.98495 |
|
2024-02-27 18:12:17,961 - INFO: Saving best model checkpoint: val_AUC 0.98454 -> 0.98495 to /app/output |
|
2024-02-27 18:12:32,189 - INFO: Training Epoch: 11 / 15 |
|
2024-02-27 18:12:32,190 - INFO: train loss: 0%| | 0/1350 [00:00<?, ?it/s] |
|
2024-02-27 18:12:32,216 - INFO: Evaluation step: 1350 |
|
2024-02-27 18:13:22,554 - INFO: train loss: 0.00: 5%|4 | 67/1350 [00:50<16:04, 1.33it/s] |
|
2024-02-27 18:13:39,767 - INFO: train loss: 0.00: 5%|4 | 67/1350 [01:07<16:04, 1.33it/s] |
|
2024-02-27 18:14:12,877 - INFO: train loss: 0.09: 10%|9 | 134/1350 [01:40<15:13, 1.33it/s] |
|
2024-02-27 18:14:29,768 - INFO: train loss: 0.09: 10%|9 | 134/1350 [01:57<15:13, 1.33it/s] |
|
2024-02-27 18:15:01,022 - INFO: train loss: 0.00: 15%|#4 | 201/1350 [02:28<14:06, 1.36it/s] |
|
2024-02-27 18:15:11,450 - INFO: train loss: 0.00: 15%|#4 | 201/1350 [02:39<14:06, 1.36it/s] |
|
2024-02-27 18:15:51,010 - INFO: train loss: 0.02: 20%|#9 | 268/1350 [03:18<13:20, 1.35it/s] |
|
2024-02-27 18:16:01,451 - INFO: train loss: 0.02: 20%|#9 | 268/1350 [03:29<13:20, 1.35it/s] |
|
2024-02-27 18:16:41,968 - INFO: train loss: 0.00: 25%|##4 | 335/1350 [04:09<12:38, 1.34it/s] |
|
2024-02-27 18:16:59,772 - INFO: train loss: 0.00: 25%|##4 | 335/1350 [04:27<12:38, 1.34it/s] |
|
2024-02-27 18:17:33,353 - INFO: train loss: 0.13: 30%|##9 | 402/1350 [05:01<11:54, 1.33it/s] |
|
2024-02-27 18:17:49,773 - INFO: train loss: 0.13: 30%|##9 | 402/1350 [05:17<11:54, 1.33it/s] |
|
2024-02-27 18:18:22,506 - INFO: train loss: 0.05: 35%|###4 | 469/1350 [05:50<10:58, 1.34it/s] |
|
2024-02-27 18:18:39,775 - INFO: train loss: 0.05: 35%|###4 | 469/1350 [06:07<10:58, 1.34it/s] |
|
2024-02-27 18:19:13,150 - INFO: train loss: 0.00: 40%|###9 | 536/1350 [06:40<10:10, 1.33it/s] |
|
2024-02-27 18:19:29,776 - INFO: train loss: 0.00: 40%|###9 | 536/1350 [06:57<10:10, 1.33it/s] |
|
2024-02-27 18:20:01,579 - INFO: train loss: 0.06: 45%|####4 | 603/1350 [07:29<09:13, 1.35it/s] |
|
2024-02-27 18:20:19,777 - INFO: train loss: 0.06: 45%|####4 | 603/1350 [07:47<09:13, 1.35it/s] |
|
2024-02-27 18:20:50,464 - INFO: train loss: 0.09: 50%|####9 | 670/1350 [08:18<08:21, 1.36it/s] |
|
2024-02-27 18:21:01,457 - INFO: train loss: 0.09: 50%|####9 | 670/1350 [08:29<08:21, 1.36it/s] |
|
2024-02-27 18:21:39,993 - INFO: train loss: 0.02: 55%|#####4 | 737/1350 [09:07<07:32, 1.35it/s] |
|
2024-02-27 18:21:50,136 - INFO: train loss: 0.02: 55%|#####4 | 737/1350 [09:17<07:32, 1.35it/s] |
|
2024-02-27 18:22:30,102 - INFO: train loss: 0.06: 60%|#####9 | 804/1350 [09:57<06:44, 1.35it/s] |
|
2024-02-27 18:22:40,137 - INFO: train loss: 0.06: 60%|#####9 | 804/1350 [10:07<06:44, 1.35it/s] |
|
2024-02-27 18:23:18,425 - INFO: train loss: 0.03: 65%|######4 | 871/1350 [10:46<05:52, 1.36it/s] |
|
2024-02-27 18:23:29,781 - INFO: train loss: 0.03: 65%|######4 | 871/1350 [10:57<05:52, 1.36it/s] |
|
2024-02-27 18:24:08,584 - INFO: train loss: 0.01: 69%|######9 | 938/1350 [11:36<05:04, 1.35it/s] |
|
2024-02-27 18:24:19,783 - INFO: train loss: 0.01: 69%|######9 | 938/1350 [11:47<05:04, 1.35it/s] |
|
2024-02-27 18:24:56,998 - INFO: train loss: 0.02: 74%|#######4 | 1005/1350 [12:24<04:13, 1.36it/s] |
|
2024-02-27 18:25:09,784 - INFO: train loss: 0.02: 74%|#######4 | 1005/1350 [12:37<04:13, 1.36it/s] |
|
2024-02-27 18:25:46,476 - INFO: train loss: 0.04: 79%|#######9 | 1072/1350 [13:14<03:24, 1.36it/s] |
|
2024-02-27 18:25:59,786 - INFO: train loss: 0.04: 79%|#######9 | 1072/1350 [13:27<03:24, 1.36it/s] |
|
2024-02-27 18:26:35,201 - INFO: train loss: 0.04: 84%|########4 | 1139/1350 [14:03<02:34, 1.36it/s] |
|
2024-02-27 18:26:49,787 - INFO: train loss: 0.04: 84%|########4 | 1139/1350 [14:17<02:34, 1.36it/s] |
|
2024-02-27 18:27:26,327 - INFO: train loss: 0.01: 89%|########9 | 1206/1350 [14:54<01:46, 1.35it/s] |
|
2024-02-27 18:27:39,788 - INFO: train loss: 0.01: 89%|########9 | 1206/1350 [15:07<01:46, 1.35it/s] |
|
2024-02-27 18:28:16,264 - INFO: train loss: 0.04: 94%|#########4| 1273/1350 [15:44<00:57, 1.35it/s] |
|
2024-02-27 18:28:29,789 - INFO: train loss: 0.04: 94%|#########4| 1273/1350 [15:57<00:57, 1.35it/s] |
|
2024-02-27 18:29:05,376 - INFO: train loss: 0.04: 99%|#########9| 1340/1350 [16:33<00:07, 1.35it/s] |
|
2024-02-27 18:29:13,213 - INFO: train loss: 0.11: 100%|##########| 1350/1350 [16:41<00:00, 1.35it/s] |
|
2024-02-27 18:29:13,218 - INFO: Starting validation inference |
|
2024-02-27 18:29:13,218 - INFO: validation progress: 0%| | 0/1200 [00:00<?, ?it/s] |
|
2024-02-27 18:29:17,580 - INFO: validation progress: 5%|5 | 60/1200 [00:04<01:22, 13.76it/s] |
|
2024-02-27 18:29:21,919 - INFO: validation progress: 10%|# | 120/1200 [00:08<01:18, 13.80it/s] |
|
2024-02-27 18:29:26,163 - INFO: validation progress: 15%|#5 | 180/1200 [00:12<01:13, 13.95it/s] |
|
2024-02-27 18:29:30,462 - INFO: validation progress: 20%|## | 240/1200 [00:17<01:08, 13.95it/s] |
|
2024-02-27 18:29:34,805 - INFO: validation progress: 25%|##5 | 300/1200 [00:21<01:04, 13.90it/s] |
|
2024-02-27 18:29:39,094 - INFO: validation progress: 30%|### | 360/1200 [00:25<01:00, 13.93it/s] |
|
2024-02-27 18:29:43,442 - INFO: validation progress: 35%|###5 | 420/1200 [00:30<00:56, 13.89it/s] |
|
2024-02-27 18:29:47,736 - INFO: validation progress: 40%|#### | 480/1200 [00:34<00:51, 13.92it/s] |
|
2024-02-27 18:29:52,079 - INFO: validation progress: 45%|####5 | 540/1200 [00:38<00:47, 13.88it/s] |
|
2024-02-27 18:29:56,392 - INFO: validation progress: 50%|##### | 600/1200 [00:43<00:43, 13.89it/s] |
|
2024-02-27 18:30:00,720 - INFO: validation progress: 55%|#####5 | 660/1200 [00:47<00:38, 13.88it/s] |
|
2024-02-27 18:30:05,044 - INFO: validation progress: 60%|###### | 720/1200 [00:51<00:34, 13.88it/s] |
|
2024-02-27 18:30:09,306 - INFO: validation progress: 65%|######5 | 780/1200 [00:56<00:30, 13.94it/s] |
|
2024-02-27 18:30:13,631 - INFO: validation progress: 70%|####### | 840/1200 [01:00<00:25, 13.92it/s] |
|
2024-02-27 18:30:17,962 - INFO: validation progress: 75%|#######5 | 900/1200 [01:04<00:21, 13.90it/s] |
|
2024-02-27 18:30:22,256 - INFO: validation progress: 80%|######## | 960/1200 [01:09<00:17, 13.92it/s] |
|
2024-02-27 18:30:26,572 - INFO: validation progress: 85%|########5 | 1020/1200 [01:13<00:12, 13.92it/s] |
|
2024-02-27 18:30:30,900 - INFO: validation progress: 90%|######### | 1080/1200 [01:17<00:08, 13.90it/s] |
|
2024-02-27 18:30:35,189 - INFO: validation progress: 95%|#########5| 1140/1200 [01:21<00:04, 13.93it/s] |
|
2024-02-27 18:30:39,433 - INFO: validation progress: 100%|##########| 1200/1200 [01:26<00:00, 13.99it/s] |
|
2024-02-27 18:30:39,438 - INFO: validation progress: 100%|##########| 1200/1200 [01:26<00:00, 13.92it/s] |
|
2024-02-27 18:30:39,467 - INFO: Mean validation loss: 0.50891 |
|
2024-02-27 18:30:39,469 - INFO: Validation AUC: 0.98125 |
|
2024-02-27 18:30:39,538 - INFO: Training Epoch: 12 / 15 |
|
2024-02-27 18:30:39,538 - INFO: train loss: 0%| | 0/1350 [00:00<?, ?it/s] |
|
2024-02-27 18:30:39,567 - INFO: Evaluation step: 1350 |
|
2024-02-27 18:31:29,498 - INFO: train loss: 0.02: 5%|4 | 67/1350 [00:49<15:56, 1.34it/s] |
|
2024-02-27 18:31:39,794 - INFO: train loss: 0.02: 5%|4 | 67/1350 [01:00<15:56, 1.34it/s] |
|
2024-02-27 18:32:17,877 - INFO: train loss: 0.07: 10%|9 | 134/1350 [01:38<14:49, 1.37it/s] |
|
2024-02-27 18:32:29,795 - INFO: train loss: 0.07: 10%|9 | 134/1350 [01:50<14:49, 1.37it/s] |
|
2024-02-27 18:33:07,379 - INFO: train loss: 0.02: 15%|#4 | 201/1350 [02:27<14:04, 1.36it/s] |
|
2024-02-27 18:33:19,796 - INFO: train loss: 0.02: 15%|#4 | 201/1350 [02:40<14:04, 1.36it/s] |
|
2024-02-27 18:33:56,250 - INFO: train loss: 0.01: 20%|#9 | 268/1350 [03:16<13:12, 1.36it/s] |
|
2024-02-27 18:34:09,798 - INFO: train loss: 0.01: 20%|#9 | 268/1350 [03:30<13:12, 1.36it/s] |
|
2024-02-27 18:34:44,070 - INFO: train loss: 0.02: 25%|##4 | 335/1350 [04:04<12:16, 1.38it/s] |
|
2024-02-27 18:34:59,799 - INFO: train loss: 0.02: 25%|##4 | 335/1350 [04:20<12:16, 1.38it/s] |
|
2024-02-27 18:35:35,083 - INFO: train loss: 0.02: 30%|##9 | 402/1350 [04:55<11:39, 1.36it/s] |
|
2024-02-27 18:35:49,800 - INFO: train loss: 0.02: 30%|##9 | 402/1350 [05:10<11:39, 1.36it/s] |
|
2024-02-27 18:36:24,126 - INFO: train loss: 0.01: 35%|###4 | 469/1350 [05:44<10:48, 1.36it/s] |
|
2024-02-27 18:36:39,801 - INFO: train loss: 0.01: 35%|###4 | 469/1350 [06:00<10:48, 1.36it/s] |
|
2024-02-27 18:37:13,111 - INFO: train loss: 0.05: 40%|###9 | 536/1350 [06:33<09:57, 1.36it/s] |
|
2024-02-27 18:37:29,803 - INFO: train loss: 0.05: 40%|###9 | 536/1350 [06:50<09:57, 1.36it/s] |
|
2024-02-27 18:38:03,364 - INFO: train loss: 0.06: 45%|####4 | 603/1350 [07:23<09:12, 1.35it/s] |
|
2024-02-27 18:38:19,804 - INFO: train loss: 0.06: 45%|####4 | 603/1350 [07:40<09:12, 1.35it/s] |
|
2024-02-27 18:38:53,347 - INFO: train loss: 0.02: 50%|####9 | 670/1350 [08:13<08:24, 1.35it/s] |
|
2024-02-27 18:39:09,805 - INFO: train loss: 0.02: 50%|####9 | 670/1350 [08:30<08:24, 1.35it/s] |
|
2024-02-27 18:39:42,089 - INFO: train loss: 0.02: 55%|#####4 | 737/1350 [09:02<07:31, 1.36it/s] |
|
2024-02-27 18:39:59,807 - INFO: train loss: 0.02: 55%|#####4 | 737/1350 [09:20<07:31, 1.36it/s] |
|
2024-02-27 18:40:33,112 - INFO: train loss: 0.06: 60%|#####9 | 804/1350 [09:53<06:46, 1.34it/s] |
|
2024-02-27 18:40:49,808 - INFO: train loss: 0.06: 60%|#####9 | 804/1350 [10:10<06:46, 1.34it/s] |
|
2024-02-27 18:41:22,702 - INFO: train loss: 0.07: 65%|######4 | 871/1350 [10:43<05:56, 1.35it/s] |
|
2024-02-27 18:41:39,809 - INFO: train loss: 0.07: 65%|######4 | 871/1350 [11:00<05:56, 1.35it/s] |
|
2024-02-27 18:42:12,320 - INFO: train loss: 0.03: 69%|######9 | 938/1350 [11:32<05:05, 1.35it/s] |
|
2024-02-27 18:42:29,811 - INFO: train loss: 0.03: 69%|######9 | 938/1350 [11:50<05:05, 1.35it/s] |
|
2024-02-27 18:43:02,020 - INFO: train loss: 0.13: 74%|#######4 | 1005/1350 [12:22<04:16, 1.35it/s] |
|
2024-02-27 18:43:19,812 - INFO: train loss: 0.13: 74%|#######4 | 1005/1350 [12:40<04:16, 1.35it/s] |
|
2024-02-27 18:43:51,520 - INFO: train loss: 0.03: 79%|#######9 | 1072/1350 [13:11<03:26, 1.35it/s] |
|
2024-02-27 18:44:09,813 - INFO: train loss: 0.03: 79%|#######9 | 1072/1350 [13:30<03:26, 1.35it/s] |
|
2024-02-27 18:44:41,522 - INFO: train loss: 0.14: 84%|########4 | 1139/1350 [14:01<02:36, 1.35it/s] |
|
2024-02-27 18:44:59,815 - INFO: train loss: 0.14: 84%|########4 | 1139/1350 [14:20<02:36, 1.35it/s] |
|
2024-02-27 18:45:32,127 - INFO: train loss: 0.06: 89%|########9 | 1206/1350 [14:52<01:47, 1.34it/s] |
|
2024-02-27 18:45:49,816 - INFO: train loss: 0.06: 89%|########9 | 1206/1350 [15:10<01:47, 1.34it/s] |
|
2024-02-27 18:46:20,702 - INFO: train loss: 0.05: 94%|#########4| 1273/1350 [15:41<00:56, 1.35it/s] |
|
2024-02-27 18:46:31,490 - INFO: train loss: 0.05: 94%|#########4| 1273/1350 [15:51<00:56, 1.35it/s] |
|
2024-02-27 18:47:08,501 - INFO: train loss: 0.06: 99%|#########9| 1340/1350 [16:28<00:07, 1.37it/s] |
|
2024-02-27 18:47:16,192 - INFO: train loss: 0.11: 100%|##########| 1350/1350 [16:36<00:00, 1.35it/s] |
|
2024-02-27 18:47:16,196 - INFO: Starting validation inference |
|
2024-02-27 18:47:16,196 - INFO: validation progress: 0%| | 0/1200 [00:00<?, ?it/s] |
|
2024-02-27 18:47:20,591 - INFO: validation progress: 5%|5 | 60/1200 [00:04<01:23, 13.65it/s] |
|
2024-02-27 18:47:24,958 - INFO: validation progress: 10%|# | 120/1200 [00:08<01:18, 13.70it/s] |
|
2024-02-27 18:47:29,240 - INFO: validation progress: 15%|#5 | 180/1200 [00:13<01:13, 13.84it/s] |
|
2024-02-27 18:47:33,560 - INFO: validation progress: 20%|## | 240/1200 [00:17<01:09, 13.86it/s] |
|
2024-02-27 18:47:37,910 - INFO: validation progress: 25%|##5 | 300/1200 [00:21<01:05, 13.84it/s] |
|
2024-02-27 18:47:42,216 - INFO: validation progress: 30%|### | 360/1200 [00:26<01:00, 13.87it/s] |
|
2024-02-27 18:47:46,563 - INFO: validation progress: 35%|###5 | 420/1200 [00:30<00:56, 13.85it/s] |
|
2024-02-27 18:47:50,871 - INFO: validation progress: 40%|#### | 480/1200 [00:34<00:51, 13.87it/s] |
|
2024-02-27 18:47:55,201 - INFO: validation progress: 45%|####5 | 540/1200 [00:39<00:47, 13.87it/s] |
|
2024-02-27 18:47:59,537 - INFO: validation progress: 50%|##### | 600/1200 [00:43<00:43, 13.86it/s] |
|
2024-02-27 18:48:03,877 - INFO: validation progress: 55%|#####5 | 660/1200 [00:47<00:38, 13.85it/s] |
|
2024-02-27 18:48:08,203 - INFO: validation progress: 60%|###### | 720/1200 [00:52<00:34, 13.85it/s] |
|
2024-02-27 18:48:12,485 - INFO: validation progress: 65%|######5 | 780/1200 [00:56<00:30, 13.90it/s] |
|
2024-02-27 18:48:16,848 - INFO: validation progress: 70%|####### | 840/1200 [01:00<00:25, 13.86it/s] |
|
2024-02-27 18:48:21,212 - INFO: validation progress: 75%|#######5 | 900/1200 [01:05<00:21, 13.82it/s] |
|
2024-02-27 18:48:25,520 - INFO: validation progress: 80%|######## | 960/1200 [01:09<00:17, 13.86it/s] |
|
2024-02-27 18:48:29,843 - INFO: validation progress: 85%|########5 | 1020/1200 [01:13<00:12, 13.86it/s] |
|
2024-02-27 18:48:34,176 - INFO: validation progress: 90%|######### | 1080/1200 [01:17<00:08, 13.86it/s] |
|
2024-02-27 18:48:38,477 - INFO: validation progress: 95%|#########5| 1140/1200 [01:22<00:04, 13.88it/s] |
|
2024-02-27 18:48:42,743 - INFO: validation progress: 100%|##########| 1200/1200 [01:26<00:00, 13.94it/s] |
|
2024-02-27 18:48:42,748 - INFO: validation progress: 100%|##########| 1200/1200 [01:26<00:00, 13.86it/s] |
|
2024-02-27 18:48:42,777 - INFO: Mean validation loss: 0.64564 |
|
2024-02-27 18:48:42,778 - INFO: Validation AUC: 0.98292 |
|
2024-02-27 18:48:42,844 - INFO: Training Epoch: 13 / 15 |
|
2024-02-27 18:48:42,845 - INFO: train loss: 0%| | 0/1350 [00:00<?, ?it/s] |
|
2024-02-27 18:48:42,873 - INFO: Evaluation step: 1350 |
|
2024-02-27 18:49:33,111 - INFO: train loss: 0.01: 5%|4 | 67/1350 [00:50<16:02, 1.33it/s] |
|
2024-02-27 18:49:49,823 - INFO: train loss: 0.01: 5%|4 | 67/1350 [01:06<16:02, 1.33it/s] |
|
2024-02-27 18:50:22,185 - INFO: train loss: 0.07: 10%|9 | 134/1350 [01:39<14:59, 1.35it/s] |
|
2024-02-27 18:50:39,825 - INFO: train loss: 0.07: 10%|9 | 134/1350 [01:56<14:59, 1.35it/s] |
|
2024-02-27 18:51:12,158 - INFO: train loss: 0.01: 15%|#4 | 201/1350 [02:29<14:13, 1.35it/s] |
|
2024-02-27 18:51:29,826 - INFO: train loss: 0.01: 15%|#4 | 201/1350 [02:46<14:13, 1.35it/s] |
|
2024-02-27 18:52:00,799 - INFO: train loss: 0.03: 20%|#9 | 268/1350 [03:17<13:16, 1.36it/s] |
|
2024-02-27 18:52:11,498 - INFO: train loss: 0.03: 20%|#9 | 268/1350 [03:28<13:16, 1.36it/s] |
|
2024-02-27 18:52:49,577 - INFO: train loss: 0.01: 25%|##4 | 335/1350 [04:06<12:24, 1.36it/s] |
|
2024-02-27 18:52:59,828 - INFO: train loss: 0.01: 25%|##4 | 335/1350 [04:16<12:24, 1.36it/s] |
|
2024-02-27 18:53:39,799 - INFO: train loss: 0.00: 30%|##9 | 402/1350 [04:56<11:40, 1.35it/s] |
|
2024-02-27 18:53:49,829 - INFO: train loss: 0.00: 30%|##9 | 402/1350 [05:06<11:40, 1.35it/s] |
|
2024-02-27 18:54:29,946 - INFO: train loss: 0.01: 35%|###4 | 469/1350 [05:47<10:53, 1.35it/s] |
|
2024-02-27 18:54:40,180 - INFO: train loss: 0.01: 35%|###4 | 469/1350 [05:57<10:53, 1.35it/s] |
|
2024-02-27 18:55:20,555 - INFO: train loss: 0.00: 40%|###9 | 536/1350 [06:37<10:07, 1.34it/s] |
|
2024-02-27 18:55:31,502 - INFO: train loss: 0.00: 40%|###9 | 536/1350 [06:48<10:07, 1.34it/s] |
|
2024-02-27 18:56:10,102 - INFO: train loss: 0.06: 45%|####4 | 603/1350 [07:27<09:15, 1.34it/s] |
|
2024-02-27 18:56:20,182 - INFO: train loss: 0.06: 45%|####4 | 603/1350 [07:37<09:15, 1.34it/s] |
|
2024-02-27 18:56:59,482 - INFO: train loss: 0.08: 50%|####9 | 670/1350 [08:16<08:24, 1.35it/s] |
|
2024-02-27 18:57:09,833 - INFO: train loss: 0.08: 50%|####9 | 670/1350 [08:26<08:24, 1.35it/s] |
|
2024-02-27 18:57:48,396 - INFO: train loss: 0.02: 55%|#####4 | 737/1350 [09:05<07:32, 1.35it/s] |
|
2024-02-27 18:57:59,835 - INFO: train loss: 0.02: 55%|#####4 | 737/1350 [09:16<07:32, 1.35it/s] |
|
2024-02-27 18:58:38,726 - INFO: train loss: 0.10: 60%|#####9 | 804/1350 [09:55<06:45, 1.35it/s] |
|
2024-02-27 18:58:49,836 - INFO: train loss: 0.10: 60%|#####9 | 804/1350 [10:06<06:45, 1.35it/s] |
|
2024-02-27 18:59:28,246 - INFO: train loss: 0.08: 65%|######4 | 871/1350 [10:45<05:55, 1.35it/s] |
|
2024-02-27 18:59:39,837 - INFO: train loss: 0.08: 65%|######4 | 871/1350 [10:56<05:55, 1.35it/s] |
|
2024-02-27 19:00:17,268 - INFO: train loss: 0.00: 69%|######9 | 938/1350 [11:34<05:04, 1.35it/s] |
|
2024-02-27 19:00:29,838 - INFO: train loss: 0.00: 69%|######9 | 938/1350 [11:46<05:04, 1.35it/s] |
|
2024-02-27 19:01:06,503 - INFO: train loss: 0.03: 74%|#######4 | 1005/1350 [12:23<04:14, 1.36it/s] |
|
2024-02-27 19:01:19,839 - INFO: train loss: 0.03: 74%|#######4 | 1005/1350 [12:36<04:14, 1.36it/s] |
|
2024-02-27 19:01:56,986 - INFO: train loss: 0.02: 79%|#######9 | 1072/1350 [13:14<03:26, 1.35it/s] |
|
2024-02-27 19:02:09,841 - INFO: train loss: 0.02: 79%|#######9 | 1072/1350 [13:26<03:26, 1.35it/s] |
|
2024-02-27 19:02:48,419 - INFO: train loss: 0.04: 84%|########4 | 1139/1350 [14:05<02:38, 1.33it/s] |
|
2024-02-27 19:02:59,842 - INFO: train loss: 0.04: 84%|########4 | 1139/1350 [14:16<02:38, 1.33it/s] |
|
2024-02-27 19:03:37,619 - INFO: train loss: 0.03: 89%|########9 | 1206/1350 [14:54<01:47, 1.34it/s] |
|
2024-02-27 19:03:49,843 - INFO: train loss: 0.03: 89%|########9 | 1206/1350 [15:06<01:47, 1.34it/s] |
|
2024-02-27 19:04:27,425 - INFO: train loss: 0.02: 94%|#########4| 1273/1350 [15:44<00:57, 1.34it/s] |
|
2024-02-27 19:04:39,844 - INFO: train loss: 0.02: 94%|#########4| 1273/1350 [15:56<00:57, 1.34it/s] |
|
2024-02-27 19:05:17,844 - INFO: train loss: 0.01: 99%|#########9| 1340/1350 [16:34<00:07, 1.34it/s] |
|
2024-02-27 19:05:25,189 - INFO: train loss: 0.02: 100%|##########| 1350/1350 [16:42<00:00, 1.35it/s] |
|
2024-02-27 19:05:25,194 - INFO: Starting validation inference |
|
2024-02-27 19:05:25,194 - INFO: validation progress: 0%| | 0/1200 [00:00<?, ?it/s] |
|
2024-02-27 19:05:29,633 - INFO: validation progress: 5%|5 | 60/1200 [00:04<01:24, 13.52it/s] |
|
2024-02-27 19:05:34,046 - INFO: validation progress: 10%|# | 120/1200 [00:08<01:19, 13.56it/s] |
|
2024-02-27 19:05:38,366 - INFO: validation progress: 15%|#5 | 180/1200 [00:13<01:14, 13.71it/s] |
|
2024-02-27 19:05:42,722 - INFO: validation progress: 20%|## | 240/1200 [00:17<01:09, 13.74it/s] |
|
2024-02-27 19:05:47,112 - INFO: validation progress: 25%|##5 | 300/1200 [00:21<01:05, 13.71it/s] |
|
2024-02-27 19:05:51,480 - INFO: validation progress: 30%|### | 360/1200 [00:26<01:01, 13.72it/s] |
|
2024-02-27 19:05:55,861 - INFO: validation progress: 35%|###5 | 420/1200 [00:30<00:56, 13.71it/s] |
|
2024-02-27 19:06:00,201 - INFO: validation progress: 40%|#### | 480/1200 [00:35<00:52, 13.75it/s] |
|
2024-02-27 19:06:04,576 - INFO: validation progress: 45%|####5 | 540/1200 [00:39<00:48, 13.74it/s] |
|
2024-02-27 19:06:08,957 - INFO: validation progress: 50%|##### | 600/1200 [00:43<00:43, 13.72it/s] |
|
2024-02-27 19:06:13,348 - INFO: validation progress: 55%|#####5 | 660/1200 [00:48<00:39, 13.71it/s] |
|
2024-02-27 19:06:17,711 - INFO: validation progress: 60%|###### | 720/1200 [00:52<00:34, 13.72it/s] |
|
2024-02-27 19:06:22,021 - INFO: validation progress: 65%|######5 | 780/1200 [00:56<00:30, 13.78it/s] |
|
2024-02-27 19:06:26,404 - INFO: validation progress: 70%|####### | 840/1200 [01:01<00:26, 13.75it/s] |
|
2024-02-27 19:06:30,790 - INFO: validation progress: 75%|#######5 | 900/1200 [01:05<00:21, 13.73it/s] |
|
2024-02-27 19:06:35,170 - INFO: validation progress: 80%|######## | 960/1200 [01:09<00:17, 13.72it/s] |
|
2024-02-27 19:06:39,540 - INFO: validation progress: 85%|########5 | 1020/1200 [01:14<00:13, 13.72it/s] |
|
2024-02-27 19:06:43,911 - INFO: validation progress: 90%|######### | 1080/1200 [01:18<00:08, 13.73it/s] |
|
2024-02-27 19:06:48,256 - INFO: validation progress: 95%|#########5| 1140/1200 [01:23<00:04, 13.75it/s] |
|
2024-02-27 19:06:52,554 - INFO: validation progress: 100%|##########| 1200/1200 [01:27<00:00, 13.81it/s] |
|
2024-02-27 19:06:52,559 - INFO: validation progress: 100%|##########| 1200/1200 [01:27<00:00, 13.74it/s] |
|
2024-02-27 19:06:52,589 - INFO: Mean validation loss: 0.62987 |
|
2024-02-27 19:06:52,590 - INFO: Validation AUC: 0.98378 |
|
2024-02-27 19:06:52,685 - INFO: Training Epoch: 14 / 15 |
|
2024-02-27 19:06:52,685 - INFO: train loss: 0%| | 0/1350 [00:00<?, ?it/s] |
|
2024-02-27 19:06:52,714 - INFO: Evaluation step: 1350 |
|
2024-02-27 19:07:42,678 - INFO: train loss: 0.00: 5%|4 | 67/1350 [00:49<15:57, 1.34it/s] |
|
2024-02-27 19:07:59,850 - INFO: train loss: 0.00: 5%|4 | 67/1350 [01:07<15:57, 1.34it/s] |
|
2024-02-27 19:08:33,409 - INFO: train loss: 0.04: 10%|9 | 134/1350 [01:40<15:15, 1.33it/s] |
|
2024-02-27 19:08:49,851 - INFO: train loss: 0.04: 10%|9 | 134/1350 [01:57<15:15, 1.33it/s] |
|
2024-02-27 19:09:23,499 - INFO: train loss: 0.10: 15%|#4 | 201/1350 [02:30<14:22, 1.33it/s] |
|
2024-02-27 19:09:39,853 - INFO: train loss: 0.10: 15%|#4 | 201/1350 [02:47<14:22, 1.33it/s] |
|
2024-02-27 19:10:12,451 - INFO: train loss: 0.06: 20%|#9 | 268/1350 [03:19<13:23, 1.35it/s] |
|
2024-02-27 19:10:29,854 - INFO: train loss: 0.06: 20%|#9 | 268/1350 [03:37<13:23, 1.35it/s] |
|
2024-02-27 19:11:00,794 - INFO: train loss: 0.02: 25%|##4 | 335/1350 [04:08<12:26, 1.36it/s] |
|
2024-02-27 19:11:11,524 - INFO: train loss: 0.02: 25%|##4 | 335/1350 [04:18<12:26, 1.36it/s] |
|
2024-02-27 19:11:50,589 - INFO: train loss: 0.00: 30%|##9 | 402/1350 [04:57<11:39, 1.36it/s] |
|
2024-02-27 19:12:01,525 - INFO: train loss: 0.00: 30%|##9 | 402/1350 [05:08<11:39, 1.36it/s] |
|
2024-02-27 19:12:41,244 - INFO: train loss: 0.02: 35%|###4 | 469/1350 [05:48<10:55, 1.34it/s] |
|
2024-02-27 19:12:51,526 - INFO: train loss: 0.02: 35%|###4 | 469/1350 [05:58<10:55, 1.34it/s] |
|
2024-02-27 19:13:29,947 - INFO: train loss: 0.00: 40%|###9 | 536/1350 [06:37<10:01, 1.35it/s] |
|
2024-02-27 19:13:40,205 - INFO: train loss: 0.00: 40%|###9 | 536/1350 [06:47<10:01, 1.35it/s] |
|
2024-02-27 19:14:19,407 - INFO: train loss: 0.02: 45%|####4 | 603/1350 [07:26<09:11, 1.35it/s] |
|
2024-02-27 19:14:29,859 - INFO: train loss: 0.02: 45%|####4 | 603/1350 [07:37<09:11, 1.35it/s] |
|
2024-02-27 19:15:10,147 - INFO: train loss: 0.14: 50%|####9 | 670/1350 [08:17<08:26, 1.34it/s] |
|
2024-02-27 19:15:20,207 - INFO: train loss: 0.14: 50%|####9 | 670/1350 [08:27<08:26, 1.34it/s] |
|
2024-02-27 19:15:58,135 - INFO: train loss: 0.02: 55%|#####4 | 737/1350 [09:05<07:30, 1.36it/s] |
|
2024-02-27 19:16:09,862 - INFO: train loss: 0.02: 55%|#####4 | 737/1350 [09:17<07:30, 1.36it/s] |
|
2024-02-27 19:16:48,384 - INFO: train loss: 0.00: 60%|#####9 | 804/1350 [09:55<06:44, 1.35it/s] |
|
2024-02-27 19:16:59,863 - INFO: train loss: 0.00: 60%|#####9 | 804/1350 [10:07<06:44, 1.35it/s] |
|
2024-02-27 19:17:38,519 - INFO: train loss: 0.08: 65%|######4 | 871/1350 [10:45<05:55, 1.35it/s] |
|
2024-02-27 19:17:49,865 - INFO: train loss: 0.08: 65%|######4 | 871/1350 [10:57<05:55, 1.35it/s] |
|
2024-02-27 19:18:27,336 - INFO: train loss: 0.04: 69%|######9 | 938/1350 [11:34<05:04, 1.35it/s] |
|
2024-02-27 19:18:39,868 - INFO: train loss: 0.04: 69%|######9 | 938/1350 [11:47<05:04, 1.35it/s] |
|
2024-02-27 19:19:19,118 - INFO: train loss: 0.04: 74%|#######4 | 1005/1350 [12:26<04:18, 1.34it/s] |
|
2024-02-27 19:19:29,870 - INFO: train loss: 0.04: 74%|#######4 | 1005/1350 [12:37<04:18, 1.34it/s] |
|
2024-02-27 19:20:09,058 - INFO: train loss: 0.06: 79%|#######9 | 1072/1350 [13:16<03:27, 1.34it/s] |
|
2024-02-27 19:20:19,871 - INFO: train loss: 0.06: 79%|#######9 | 1072/1350 [13:27<03:27, 1.34it/s] |
|
2024-02-27 19:20:58,304 - INFO: train loss: 0.01: 84%|########4 | 1139/1350 [14:05<02:36, 1.34it/s] |
|
2024-02-27 19:21:09,872 - INFO: train loss: 0.01: 84%|########4 | 1139/1350 [14:17<02:36, 1.34it/s] |
|
2024-02-27 19:21:47,297 - INFO: train loss: 0.05: 89%|########9 | 1206/1350 [14:54<01:46, 1.35it/s] |
|
2024-02-27 19:21:59,873 - INFO: train loss: 0.05: 89%|########9 | 1206/1350 [15:07<01:46, 1.35it/s] |
|
2024-02-27 19:22:37,073 - INFO: train loss: 0.03: 94%|#########4| 1273/1350 [15:44<00:57, 1.35it/s] |
|
2024-02-27 19:22:49,875 - INFO: train loss: 0.03: 94%|#########4| 1273/1350 [15:57<00:57, 1.35it/s] |
|
2024-02-27 19:23:26,790 - INFO: train loss: 0.01: 99%|#########9| 1340/1350 [16:34<00:07, 1.35it/s] |
|
2024-02-27 19:23:34,256 - INFO: train loss: 0.01: 100%|##########| 1350/1350 [16:41<00:00, 1.35it/s] |
|
2024-02-27 19:23:34,260 - INFO: Starting validation inference |
|
2024-02-27 19:23:34,261 - INFO: validation progress: 0%| | 0/1200 [00:00<?, ?it/s] |
|
2024-02-27 19:23:38,716 - INFO: validation progress: 5%|5 | 60/1200 [00:04<01:24, 13.47it/s] |
|
2024-02-27 19:23:43,151 - INFO: validation progress: 10%|# | 120/1200 [00:08<01:19, 13.50it/s] |
|
2024-02-27 19:23:47,507 - INFO: validation progress: 15%|#5 | 180/1200 [00:13<01:14, 13.63it/s] |
|
2024-02-27 19:23:51,895 - INFO: validation progress: 20%|## | 240/1200 [00:17<01:10, 13.64it/s] |
|
2024-02-27 19:23:56,324 - INFO: validation progress: 25%|##5 | 300/1200 [00:22<01:06, 13.61it/s] |
|
2024-02-27 19:24:00,710 - INFO: validation progress: 30%|### | 360/1200 [00:26<01:01, 13.63it/s] |
|
2024-02-27 19:24:05,134 - INFO: validation progress: 35%|###5 | 420/1200 [00:30<00:57, 13.61it/s] |
|
2024-02-27 19:24:09,515 - INFO: validation progress: 40%|#### | 480/1200 [00:35<00:52, 13.64it/s] |
|
2024-02-27 19:24:13,927 - INFO: validation progress: 45%|####5 | 540/1200 [00:39<00:48, 13.62it/s] |
|
2024-02-27 19:24:18,336 - INFO: validation progress: 50%|##### | 600/1200 [00:44<00:44, 13.62it/s] |
|
2024-02-27 19:24:22,752 - INFO: validation progress: 55%|#####5 | 660/1200 [00:48<00:39, 13.61it/s] |
|
2024-02-27 19:24:27,156 - INFO: validation progress: 60%|###### | 720/1200 [00:52<00:35, 13.61it/s] |
|
2024-02-27 19:24:31,513 - INFO: validation progress: 65%|######5 | 780/1200 [00:57<00:30, 13.66it/s] |
|
2024-02-27 19:24:35,914 - INFO: validation progress: 70%|####### | 840/1200 [01:01<00:26, 13.65it/s] |
|
2024-02-27 19:24:40,325 - INFO: validation progress: 75%|#######5 | 900/1200 [01:06<00:21, 13.64it/s] |
|
2024-02-27 19:24:44,716 - INFO: validation progress: 80%|######## | 960/1200 [01:10<00:17, 13.65it/s] |
|
2024-02-27 19:24:49,117 - INFO: validation progress: 85%|########5 | 1020/1200 [01:14<00:13, 13.64it/s] |
|
2024-02-27 19:24:53,534 - INFO: validation progress: 90%|######### | 1080/1200 [01:19<00:08, 13.62it/s] |
|
2024-02-27 19:24:57,899 - INFO: validation progress: 95%|#########5| 1140/1200 [01:23<00:04, 13.66it/s] |
|
2024-02-27 19:25:02,223 - INFO: validation progress: 100%|##########| 1200/1200 [01:27<00:00, 13.73it/s] |
|
2024-02-27 19:25:02,229 - INFO: validation progress: 100%|##########| 1200/1200 [01:27<00:00, 13.64it/s] |
|
2024-02-27 19:25:02,257 - INFO: Mean validation loss: 0.74412 |
|
2024-02-27 19:25:02,259 - INFO: Validation AUC: 0.98246 |
|
2024-02-27 19:25:02,327 - INFO: Training Epoch: 15 / 15 |
|
2024-02-27 19:25:02,327 - INFO: train loss: 0%| | 0/1350 [00:00<?, ?it/s] |
|
2024-02-27 19:25:02,355 - INFO: Evaluation step: 1350 |
|
2024-02-27 19:25:52,461 - INFO: train loss: 0.00: 5%|4 | 67/1350 [00:50<16:00, 1.34it/s] |
|
2024-02-27 19:26:09,879 - INFO: train loss: 0.00: 5%|4 | 67/1350 [01:07<16:00, 1.34it/s] |
|
2024-02-27 19:26:42,728 - INFO: train loss: 0.08: 10%|9 | 134/1350 [01:40<15:11, 1.33it/s] |
|
2024-02-27 19:26:59,880 - INFO: train loss: 0.08: 10%|9 | 134/1350 [01:57<15:11, 1.33it/s] |
|
2024-02-27 19:27:31,814 - INFO: train loss: 0.00: 15%|#4 | 201/1350 [02:29<14:12, 1.35it/s] |
|
2024-02-27 19:27:49,881 - INFO: train loss: 0.00: 15%|#4 | 201/1350 [02:47<14:12, 1.35it/s] |
|
2024-02-27 19:28:20,956 - INFO: train loss: 0.03: 20%|#9 | 268/1350 [03:18<13:19, 1.35it/s] |
|
2024-02-27 19:28:31,548 - INFO: train loss: 0.03: 20%|#9 | 268/1350 [03:29<13:19, 1.35it/s] |
|
2024-02-27 19:29:10,893 - INFO: train loss: 0.23: 25%|##4 | 335/1350 [04:08<12:32, 1.35it/s] |
|
2024-02-27 19:29:21,549 - INFO: train loss: 0.23: 25%|##4 | 335/1350 [04:19<12:32, 1.35it/s] |
|
2024-02-27 19:30:01,674 - INFO: train loss: 0.03: 30%|##9 | 402/1350 [04:59<11:47, 1.34it/s] |
|
2024-02-27 19:30:19,884 - INFO: train loss: 0.03: 30%|##9 | 402/1350 [05:17<11:47, 1.34it/s] |
|
2024-02-27 19:30:49,522 - INFO: train loss: 0.14: 35%|###4 | 469/1350 [05:47<10:48, 1.36it/s] |
|
2024-02-27 19:30:59,885 - INFO: train loss: 0.14: 35%|###4 | 469/1350 [05:57<10:48, 1.36it/s] |
|
2024-02-27 19:31:39,371 - INFO: train loss: 0.01: 40%|###9 | 536/1350 [06:37<10:01, 1.35it/s] |
|
2024-02-27 19:31:49,886 - INFO: train loss: 0.01: 40%|###9 | 536/1350 [06:47<10:01, 1.35it/s] |
|
2024-02-27 19:32:29,545 - INFO: train loss: 0.01: 45%|####4 | 603/1350 [07:27<09:14, 1.35it/s] |
|
2024-02-27 19:32:39,887 - INFO: train loss: 0.01: 45%|####4 | 603/1350 [07:37<09:14, 1.35it/s] |
|
2024-02-27 19:33:19,844 - INFO: train loss: 0.12: 50%|####9 | 670/1350 [08:17<08:26, 1.34it/s] |
|
2024-02-27 19:33:29,888 - INFO: train loss: 0.12: 50%|####9 | 670/1350 [08:27<08:26, 1.34it/s] |
|
2024-02-27 19:34:08,989 - INFO: train loss: 0.03: 55%|#####4 | 737/1350 [09:06<07:34, 1.35it/s] |
|
2024-02-27 19:34:19,889 - INFO: train loss: 0.03: 55%|#####4 | 737/1350 [09:17<07:34, 1.35it/s] |
|
2024-02-27 19:34:58,941 - INFO: train loss: 0.01: 60%|#####9 | 804/1350 [09:56<06:45, 1.35it/s] |
|
2024-02-27 19:35:09,891 - INFO: train loss: 0.01: 60%|#####9 | 804/1350 [10:07<06:45, 1.35it/s] |
|
2024-02-27 19:35:49,846 - INFO: train loss: 0.01: 65%|######4 | 871/1350 [10:47<05:58, 1.34it/s] |
|
2024-02-27 19:35:59,892 - INFO: train loss: 0.01: 65%|######4 | 871/1350 [10:57<05:58, 1.34it/s] |
|
2024-02-27 19:36:37,800 - INFO: train loss: 0.02: 69%|######9 | 938/1350 [11:35<05:04, 1.35it/s] |
|
2024-02-27 19:36:49,893 - INFO: train loss: 0.02: 69%|######9 | 938/1350 [11:47<05:04, 1.35it/s] |
|
2024-02-27 19:37:27,177 - INFO: train loss: 0.02: 74%|#######4 | 1005/1350 [12:24<04:14, 1.36it/s] |
|
2024-02-27 19:37:39,894 - INFO: train loss: 0.02: 74%|#######4 | 1005/1350 [12:37<04:14, 1.36it/s] |
|
2024-02-27 19:38:18,232 - INFO: train loss: 0.02: 79%|#######9 | 1072/1350 [13:15<03:27, 1.34it/s] |
|
2024-02-27 19:38:29,895 - INFO: train loss: 0.02: 79%|#######9 | 1072/1350 [13:27<03:27, 1.34it/s] |
|
2024-02-27 19:39:07,223 - INFO: train loss: 0.13: 84%|########4 | 1139/1350 [14:04<02:36, 1.35it/s] |
|
2024-02-27 19:39:19,896 - INFO: train loss: 0.13: 84%|########4 | 1139/1350 [14:17<02:36, 1.35it/s] |
|
2024-02-27 19:39:58,424 - INFO: train loss: 0.00: 89%|########9 | 1206/1350 [14:56<01:47, 1.34it/s] |
|
2024-02-27 19:40:09,897 - INFO: train loss: 0.00: 89%|########9 | 1206/1350 [15:07<01:47, 1.34it/s] |
|
2024-02-27 19:40:48,328 - INFO: train loss: 0.01: 94%|#########4| 1273/1350 [15:46<00:57, 1.34it/s] |
|
2024-02-27 19:40:59,899 - INFO: train loss: 0.01: 94%|#########4| 1273/1350 [15:57<00:57, 1.34it/s] |
|
2024-02-27 19:41:38,376 - INFO: train loss: 0.08: 99%|#########9| 1340/1350 [16:36<00:07, 1.34it/s] |
|
2024-02-27 19:41:45,770 - INFO: train loss: 0.05: 100%|##########| 1350/1350 [16:43<00:00, 1.35it/s] |
|
2024-02-27 19:41:45,775 - INFO: Starting validation inference |
|
2024-02-27 19:41:45,775 - INFO: validation progress: 0%| | 0/1200 [00:00<?, ?it/s] |
|
2024-02-27 19:41:50,265 - INFO: validation progress: 5%|5 | 60/1200 [00:04<01:25, 13.36it/s] |
|
2024-02-27 19:41:54,699 - INFO: validation progress: 10%|# | 120/1200 [00:08<01:20, 13.46it/s] |
|
2024-02-27 19:41:59,047 - INFO: validation progress: 15%|#5 | 180/1200 [00:13<01:14, 13.61it/s] |
|
2024-02-27 19:42:03,453 - INFO: validation progress: 20%|## | 240/1200 [00:17<01:10, 13.62it/s] |
|
2024-02-27 19:42:07,886 - INFO: validation progress: 25%|##5 | 300/1200 [00:22<01:06, 13.59it/s] |
|
2024-02-27 19:42:12,291 - INFO: validation progress: 30%|### | 360/1200 [00:26<01:01, 13.60it/s] |
|
2024-02-27 19:42:16,735 - INFO: validation progress: 35%|###5 | 420/1200 [00:30<00:57, 13.57it/s] |
|
2024-02-27 19:42:21,142 - INFO: validation progress: 40%|#### | 480/1200 [00:35<00:53, 13.58it/s] |
|
2024-02-27 19:42:25,564 - INFO: validation progress: 45%|####5 | 540/1200 [00:39<00:48, 13.58it/s] |
|
2024-02-27 19:42:29,989 - INFO: validation progress: 50%|##### | 600/1200 [00:44<00:44, 13.57it/s] |
|
2024-02-27 19:42:34,421 - INFO: validation progress: 55%|#####5 | 660/1200 [00:48<00:39, 13.56it/s] |
|
2024-02-27 19:42:38,848 - INFO: validation progress: 60%|###### | 720/1200 [00:53<00:35, 13.56it/s] |
|
2024-02-27 19:42:43,219 - INFO: validation progress: 65%|######5 | 780/1200 [00:57<00:30, 13.61it/s] |
|
2024-02-27 19:42:47,653 - INFO: validation progress: 70%|####### | 840/1200 [01:01<00:26, 13.59it/s] |
|
2024-02-27 19:42:52,078 - INFO: validation progress: 75%|#######5 | 900/1200 [01:06<00:22, 13.58it/s] |
|
2024-02-27 19:42:56,490 - INFO: validation progress: 80%|######## | 960/1200 [01:10<00:17, 13.58it/s] |
|
2024-02-27 19:43:00,920 - INFO: validation progress: 85%|########5 | 1020/1200 [01:15<00:13, 13.57it/s] |
|
2024-02-27 19:43:05,362 - INFO: validation progress: 90%|######### | 1080/1200 [01:19<00:08, 13.55it/s] |
|
2024-02-27 19:43:09,758 - INFO: validation progress: 95%|#########5| 1140/1200 [01:23<00:04, 13.58it/s] |
|
2024-02-27 19:43:14,101 - INFO: validation progress: 100%|##########| 1200/1200 [01:28<00:00, 13.65it/s] |
|
2024-02-27 19:43:14,108 - INFO: validation progress: 100%|##########| 1200/1200 [01:28<00:00, 13.58it/s] |
|
2024-02-27 19:43:14,136 - INFO: Mean validation loss: 0.74600 |
|
2024-02-27 19:43:14,138 - INFO: Validation AUC: 0.98278 |
|
|
