2023-09-04 02:32:18,578 44k INFO {'train': {'log_interval': 200, 'eval_interval': 800, 'seed': 1234, 'epochs': 10000, 'learning_rate': 0.0001, 'betas': [0.8, 0.99], 'eps': 1e-09, 'batch_size': 24, 'fp16_run': True, 'half_type': 'fp16', 'lr_decay': 0.999875, 'segment_size': 10240, 'init_lr_ratio': 1, 'warmup_epochs': 0, 'c_mel': 45, 'c_kl': 1.0, 'use_sr': True, 'max_speclen': 512, 'port': '8001', 'keep_ckpts': 3, 'all_in_mem': True, 'vol_aug': True}, 'data': {'training_files': 'filelists/train.txt', 'validation_files': 'filelists/val.txt', 'max_wav_value': 32768.0, 'sampling_rate': 44100, 'filter_length': 2048, 'hop_length': 512, 'win_length': 2048, 'n_mel_channels': 80, 'mel_fmin': 0.0, 'mel_fmax': 22050, 'unit_interpolate_mode': 'nearest'}, 'model': {'inter_channels': 192, 'hidden_channels': 192, 'filter_channels': 768, 'n_heads': 2, 'n_layers': 6, 'kernel_size': 3, 'p_dropout': 0.1, 'resblock': '1', 'resblock_kernel_sizes': [3, 7, 11], 'resblock_dilation_sizes': [[1, 3, 5], [1, 3, 5], [1, 3, 5]], 'upsample_rates': [8, 8, 2, 2, 2], 'upsample_initial_channel': 512, 'upsample_kernel_sizes': [16, 16, 4, 4, 4], 'n_layers_q': 3, 'n_layers_trans_flow': 3, 'n_flow_layer': 4, 'use_spectral_norm': False, 'gin_channels': 768, 'ssl_dim': 768, 'n_speakers': 1, 'vocoder_name': 'nsf-hifigan', 'speech_encoder': 'vec768l12', 'speaker_embedding': False, 'vol_embedding': True, 'use_depthwise_conv': False, 'flow_share_parameter': False, 'use_automatic_f0_prediction': True, 'use_transformer_flow': False}, 'spk': {'AgnesTachyon': 0}, 'model_dir': './logs\\44k'} 2023-09-04 02:32:18,578 44k WARNING G:\Sound Convertion\so-vits-svc-4.1-Fork is not a git repository, therefore hash value comparison will be ignored. 2023-09-04 02:32:29,367 44k INFO emb_g.weight is not in the checkpoint 2023-09-04 02:32:29,408 44k INFO Loaded checkpoint './logs\44k\G_0.pth' (iteration 0) 2023-09-04 02:32:29,955 44k INFO Loaded checkpoint './logs\44k\D_0.pth' (iteration 0) 2023-09-04 02:33:01,012 44k INFO ====> Epoch: 1, cost 42.44 s 2023-09-04 02:33:16,682 44k INFO ====> Epoch: 2, cost 15.67 s 2023-09-04 02:33:32,318 44k INFO ====> Epoch: 3, cost 15.64 s 2023-09-04 02:33:49,679 44k INFO ====> Epoch: 4, cost 17.36 s 2023-09-04 02:34:05,041 44k INFO ====> Epoch: 5, cost 15.36 s 2023-09-04 02:34:20,743 44k INFO ====> Epoch: 6, cost 15.70 s 2023-09-04 02:34:35,853 44k INFO ====> Epoch: 7, cost 15.11 s 2023-09-04 02:34:51,535 44k INFO ====> Epoch: 8, cost 15.68 s 2023-09-04 02:35:07,717 44k INFO ====> Epoch: 9, cost 16.18 s 2023-09-04 02:35:16,452 44k INFO Train Epoch: 10 [48%] 2023-09-04 02:35:16,452 44k INFO Losses: [2.393909215927124, 2.6556153297424316, 7.531515598297119, 18.026809692382812, 1.3180642127990723], step: 200, lr: 9.98875562335968e-05, reference_loss: 31.925912857055664 2023-09-04 02:35:24,293 44k INFO ====> Epoch: 10, cost 16.58 s 2023-09-04 02:35:40,252 44k INFO ====> Epoch: 11, cost 15.96 s 2023-09-04 02:35:56,096 44k INFO ====> Epoch: 12, cost 15.84 s 2023-09-04 02:36:12,134 44k INFO ====> Epoch: 13, cost 16.04 s 2023-09-04 02:36:28,114 44k INFO ====> Epoch: 14, cost 15.98 s 2023-09-04 02:36:43,935 44k INFO ====> Epoch: 15, cost 15.82 s 2023-09-04 02:36:59,543 44k INFO ====> Epoch: 16, cost 15.61 s 2023-09-04 02:37:14,999 44k INFO ====> Epoch: 17, cost 15.46 s 2023-09-04 02:37:30,094 44k INFO ====> Epoch: 18, cost 15.09 s 2023-09-04 02:37:45,364 44k INFO ====> Epoch: 19, cost 15.27 s 2023-09-04 02:37:46,190 44k INFO Train Epoch: 20 [0%] 2023-09-04 02:37:46,191 44k INFO Losses: [2.39064359664917, 2.7290825843811035, 8.186314582824707, 18.993837356567383, 1.2159228324890137], step: 400, lr: 9.976276699833672e-05, reference_loss: 33.51580047607422 2023-09-04 02:38:01,779 44k INFO ====> Epoch: 20, cost 16.41 s 2023-09-04 02:38:16,943 44k INFO ====> Epoch: 21, cost 15.16 s 2023-09-04 02:38:31,967 44k INFO ====> Epoch: 22, cost 15.02 s 2023-09-04 02:38:47,013 44k INFO ====> Epoch: 23, cost 15.05 s 2023-09-04 02:39:01,748 44k INFO ====> Epoch: 24, cost 14.73 s 2023-09-04 02:39:16,844 44k INFO ====> Epoch: 25, cost 15.10 s 2023-09-04 02:39:31,566 44k INFO ====> Epoch: 26, cost 14.72 s 2023-09-04 02:39:46,310 44k INFO ====> Epoch: 27, cost 14.74 s 2023-09-04 02:40:01,317 44k INFO ====> Epoch: 28, cost 15.01 s 2023-09-04 02:40:09,989 44k INFO Train Epoch: 29 [52%] 2023-09-04 02:40:09,989 44k INFO Losses: [2.5193567276000977, 2.3155932426452637, 7.788388252258301, 20.070510864257812, 1.1859819889068604], step: 600, lr: 9.965058998565574e-05, reference_loss: 33.87982940673828 2023-09-04 02:40:16,469 44k INFO ====> Epoch: 29, cost 15.15 s 2023-09-04 02:40:31,004 44k INFO ====> Epoch: 30, cost 14.53 s 2023-09-04 02:40:45,777 44k INFO ====> Epoch: 31, cost 14.77 s 2023-09-04 02:41:00,255 44k INFO ====> Epoch: 32, cost 14.48 s 2023-09-04 02:41:14,977 44k INFO ====> Epoch: 33, cost 14.72 s 2023-09-04 02:41:29,680 44k INFO ====> Epoch: 34, cost 14.70 s 2023-09-04 02:41:44,464 44k INFO ====> Epoch: 35, cost 14.78 s 2023-09-04 02:41:59,198 44k INFO ====> Epoch: 36, cost 14.73 s 2023-09-04 02:42:13,802 44k INFO ====> Epoch: 37, cost 14.60 s 2023-09-04 02:42:30,128 44k INFO ====> Epoch: 38, cost 16.33 s 2023-09-04 02:42:31,747 44k INFO Train Epoch: 39 [5%] 2023-09-04 02:42:31,747 44k INFO Losses: [2.4047131538391113, 2.5111870765686035, 7.487963676452637, 18.751880645751953, 0.9878390431404114], step: 800, lr: 9.952609679164422e-05, reference_loss: 32.143585205078125 2023-09-04 02:42:46,246 44k INFO Saving model and optimizer state at iteration 39 to ./logs\44k\G_800.pth 2023-09-04 02:42:47,123 44k INFO Saving model and optimizer state at iteration 39 to ./logs\44k\D_800.pth 2023-09-04 02:43:03,298 44k INFO ====> Epoch: 39, cost 33.17 s 2023-09-04 02:43:18,071 44k INFO ====> Epoch: 40, cost 14.77 s 2023-09-04 02:43:32,838 44k INFO ====> Epoch: 41, cost 14.77 s 2023-09-04 02:43:47,614 44k INFO ====> Epoch: 42, cost 14.78 s 2023-09-04 02:44:02,635 44k INFO ====> Epoch: 43, cost 15.02 s 2023-09-04 02:44:17,454 44k INFO ====> Epoch: 44, cost 14.82 s 2023-09-04 02:44:32,417 44k INFO ====> Epoch: 45, cost 14.96 s 2023-09-04 02:44:47,319 44k INFO ====> Epoch: 46, cost 14.90 s 2023-09-04 02:45:02,358 44k INFO ====> Epoch: 47, cost 15.04 s 2023-09-04 02:45:11,837 44k INFO Train Epoch: 48 [57%] 2023-09-04 02:45:11,837 44k INFO Losses: [2.3672678470611572, 2.5863869190216064, 6.757925033569336, 17.29076385498047, 1.063711166381836], step: 1000, lr: 9.941418589985758e-05, reference_loss: 30.066055297851562 2023-09-04 02:45:17,784 44k INFO ====> Epoch: 48, cost 15.43 s 2023-09-04 02:45:33,492 44k INFO ====> Epoch: 49, cost 15.71 s 2023-09-04 02:45:48,886 44k INFO ====> Epoch: 50, cost 15.39 s 2023-09-04 02:46:03,866 44k INFO ====> Epoch: 51, cost 14.98 s 2023-09-04 02:46:18,859 44k INFO ====> Epoch: 52, cost 14.99 s 2023-09-04 02:46:33,819 44k INFO ====> Epoch: 53, cost 14.96 s 2023-09-04 02:46:48,754 44k INFO ====> Epoch: 54, cost 14.94 s 2023-09-04 02:47:03,517 44k INFO ====> Epoch: 55, cost 14.76 s 2023-09-04 02:47:18,435 44k INFO ====> Epoch: 56, cost 14.92 s 2023-09-04 02:47:33,538 44k INFO ====> Epoch: 57, cost 15.10 s 2023-09-04 02:47:35,728 44k INFO Train Epoch: 58 [10%] 2023-09-04 02:47:35,729 44k INFO Losses: [2.4853034019470215, 2.3893792629241943, 6.7385125160217285, 17.71141815185547, 0.9717617034912109], step: 1200, lr: 9.928998804478705e-05, reference_loss: 30.296375274658203 2023-09-04 02:47:48,724 44k INFO ====> Epoch: 58, cost 15.19 s 2023-09-04 02:48:03,717 44k INFO ====> Epoch: 59, cost 14.99 s 2023-09-04 02:48:18,626 44k INFO ====> Epoch: 60, cost 14.91 s 2023-09-04 02:48:33,294 44k INFO ====> Epoch: 61, cost 14.67 s 2023-09-04 02:48:48,204 44k INFO ====> Epoch: 62, cost 14.91 s 2023-09-04 02:49:02,807 44k INFO ====> Epoch: 63, cost 14.60 s 2023-09-04 02:49:18,739 44k INFO ====> Epoch: 64, cost 15.93 s 2023-09-04 02:49:33,437 44k INFO ====> Epoch: 65, cost 14.70 s 2023-09-04 02:49:48,388 44k INFO ====> Epoch: 66, cost 14.95 s 2023-09-04 02:49:58,433 44k INFO Train Epoch: 67 [62%] 2023-09-04 02:49:58,434 44k INFO Losses: [2.386770486831665, 2.3963820934295654, 6.418186187744141, 17.654911041259766, 0.9085859656333923], step: 1400, lr: 9.917834264256819e-05, reference_loss: 29.76483726501465 2023-09-04 02:50:03,509 44k INFO ====> Epoch: 67, cost 15.12 s 2023-09-04 02:50:18,185 44k INFO ====> Epoch: 68, cost 14.68 s 2023-09-04 02:50:32,858 44k INFO ====> Epoch: 69, cost 14.67 s 2023-09-04 02:50:47,913 44k INFO ====> Epoch: 70, cost 15.06 s 2023-09-04 02:51:02,704 44k INFO ====> Epoch: 71, cost 14.79 s 2023-09-04 02:51:17,533 44k INFO ====> Epoch: 72, cost 14.83 s 2023-09-04 02:51:32,263 44k INFO ====> Epoch: 73, cost 14.73 s 2023-09-04 02:51:47,048 44k INFO ====> Epoch: 74, cost 14.78 s 2023-09-04 02:52:01,601 44k INFO ====> Epoch: 75, cost 14.55 s 2023-09-04 02:52:16,067 44k INFO ====> Epoch: 76, cost 14.47 s 2023-09-04 02:52:18,844 44k INFO Train Epoch: 77 [14%] 2023-09-04 02:52:18,845 44k INFO Losses: [2.624443292617798, 2.4071242809295654, 7.505742073059082, 18.465816497802734, 1.075671672821045], step: 1600, lr: 9.905443942579728e-05, reference_loss: 32.07879638671875 2023-09-04 02:52:24,625 44k INFO Saving model and optimizer state at iteration 77 to ./logs\44k\G_1600.pth 2023-09-04 02:52:25,263 44k INFO Saving model and optimizer state at iteration 77 to ./logs\44k\D_1600.pth 2023-09-04 02:52:38,113 44k INFO ====> Epoch: 77, cost 22.05 s 2023-09-04 02:52:52,779 44k INFO ====> Epoch: 78, cost 14.67 s 2023-09-04 02:53:07,315 44k INFO ====> Epoch: 79, cost 14.54 s 2023-09-04 02:53:21,731 44k INFO ====> Epoch: 80, cost 14.42 s 2023-09-04 02:53:36,226 44k INFO ====> Epoch: 81, cost 14.49 s 2023-09-04 02:53:50,882 44k INFO ====> Epoch: 82, cost 14.66 s 2023-09-04 02:54:05,466 44k INFO ====> Epoch: 83, cost 14.58 s 2023-09-04 02:54:19,939 44k INFO ====> Epoch: 84, cost 14.47 s 2023-09-04 02:54:34,321 44k INFO ====> Epoch: 85, cost 14.38 s 2023-09-04 02:54:44,856 44k INFO Train Epoch: 86 [67%] 2023-09-04 02:54:44,857 44k INFO Losses: [2.3498263359069824, 2.2830097675323486, 5.864878177642822, 16.610010147094727, 0.9492321014404297], step: 1800, lr: 9.894305888331732e-05, reference_loss: 28.056957244873047 2023-09-04 02:54:49,211 44k INFO ====> Epoch: 86, cost 14.89 s 2023-09-04 02:55:03,874 44k INFO ====> Epoch: 87, cost 14.66 s 2023-09-04 02:55:18,104 44k INFO ====> Epoch: 88, cost 14.23 s 2023-09-04 02:55:32,553 44k INFO ====> Epoch: 89, cost 14.45 s 2023-09-04 02:55:47,012 44k INFO ====> Epoch: 90, cost 14.46 s 2023-09-04 02:56:01,672 44k INFO ====> Epoch: 91, cost 14.66 s 2023-09-04 02:56:16,026 44k INFO ====> Epoch: 92, cost 14.35 s 2023-09-04 02:56:30,336 44k INFO ====> Epoch: 93, cost 14.31 s 2023-09-04 02:56:44,746 44k INFO ====> Epoch: 94, cost 14.41 s 2023-09-04 02:56:59,117 44k INFO ====> Epoch: 95, cost 14.37 s 2023-09-04 02:57:02,630 44k INFO Train Epoch: 96 [19%] 2023-09-04 02:57:02,630 44k INFO Losses: [2.5019257068634033, 2.3453621864318848, 6.7265305519104, 18.371368408203125, 0.9076389074325562], step: 2000, lr: 9.881944960586671e-05, reference_loss: 30.852825164794922 2023-09-04 02:57:13,874 44k INFO ====> Epoch: 96, cost 14.76 s 2023-09-04 02:57:28,193 44k INFO ====> Epoch: 97, cost 14.32 s 2023-09-04 02:57:42,748 44k INFO ====> Epoch: 98, cost 14.56 s 2023-09-04 02:57:57,241 44k INFO ====> Epoch: 99, cost 14.49 s 2023-09-04 02:58:11,812 44k INFO ====> Epoch: 100, cost 14.57 s 2023-09-04 02:58:26,296 44k INFO ====> Epoch: 101, cost 14.48 s 2023-09-04 02:58:40,945 44k INFO ====> Epoch: 102, cost 14.65 s 2023-09-04 02:58:55,369 44k INFO ====> Epoch: 103, cost 14.42 s 2023-09-04 02:59:09,982 44k INFO ====> Epoch: 104, cost 14.61 s 2023-09-04 02:59:21,272 44k INFO Train Epoch: 105 [71%] 2023-09-04 02:59:21,272 44k INFO Losses: [2.6059176921844482, 2.2382233142852783, 5.262292385101318, 16.338266372680664, 0.7635509371757507], step: 2200, lr: 9.870833329479095e-05, reference_loss: 27.208251953125 2023-09-04 02:59:24,795 44k INFO ====> Epoch: 105, cost 14.81 s 2023-09-04 02:59:39,310 44k INFO ====> Epoch: 106, cost 14.51 s 2023-09-04 02:59:53,870 44k INFO ====> Epoch: 107, cost 14.56 s 2023-09-04 03:00:08,442 44k INFO ====> Epoch: 108, cost 14.57 s 2023-09-04 03:00:22,896 44k INFO ====> Epoch: 109, cost 14.45 s 2023-09-04 03:00:37,424 44k INFO ====> Epoch: 110, cost 14.53 s 2023-09-04 03:00:51,875 44k INFO ====> Epoch: 111, cost 14.45 s 2023-09-04 03:01:06,315 44k INFO ====> Epoch: 112, cost 14.44 s 2023-09-04 03:01:20,834 44k INFO ====> Epoch: 113, cost 14.52 s 2023-09-04 03:01:35,272 44k INFO ====> Epoch: 114, cost 14.44 s 2023-09-04 03:01:39,461 44k INFO Train Epoch: 115 [24%] 2023-09-04 03:01:39,462 44k INFO Losses: [2.66475772857666, 2.099271059036255, 6.341681957244873, 17.710691452026367, 1.0697828531265259], step: 2400, lr: 9.858501725933955e-05, reference_loss: 29.886184692382812 2023-09-04 03:01:44,999 44k INFO Saving model and optimizer state at iteration 115 to ./logs\44k\G_2400.pth 2023-09-04 03:01:45,568 44k INFO Saving model and optimizer state at iteration 115 to ./logs\44k\D_2400.pth 2023-09-04 03:01:57,621 44k INFO ====> Epoch: 115, cost 22.35 s 2023-09-04 03:02:12,106 44k INFO ====> Epoch: 116, cost 14.49 s 2023-09-04 03:02:26,686 44k INFO ====> Epoch: 117, cost 14.58 s 2023-09-04 03:02:41,078 44k INFO ====> Epoch: 118, cost 14.39 s 2023-09-04 03:02:55,754 44k INFO ====> Epoch: 119, cost 14.68 s 2023-09-04 03:03:10,231 44k INFO ====> Epoch: 120, cost 14.48 s 2023-09-04 03:03:24,532 44k INFO ====> Epoch: 121, cost 14.30 s 2023-09-04 03:03:39,122 44k INFO ====> Epoch: 122, cost 14.59 s 2023-09-04 03:03:53,694 44k INFO ====> Epoch: 123, cost 14.57 s 2023-09-04 03:04:05,722 44k INFO Train Epoch: 124 [76%] 2023-09-04 03:04:05,722 44k INFO Losses: [2.3513760566711426, 2.574720859527588, 7.756329536437988, 20.17890167236328, 1.0951282978057861], step: 2600, lr: 9.847416455282387e-05, reference_loss: 33.95645523071289 2023-09-04 03:04:08,588 44k INFO ====> Epoch: 124, cost 14.89 s 2023-09-04 03:04:23,318 44k INFO ====> Epoch: 125, cost 14.73 s 2023-09-04 03:04:37,744 44k INFO ====> Epoch: 126, cost 14.43 s 2023-09-04 03:04:52,379 44k INFO ====> Epoch: 127, cost 14.63 s 2023-09-04 03:05:07,198 44k INFO ====> Epoch: 128, cost 14.82 s 2023-09-04 03:05:21,669 44k INFO ====> Epoch: 129, cost 14.47 s 2023-09-04 03:05:36,023 44k INFO ====> Epoch: 130, cost 14.35 s 2023-09-04 03:05:50,856 44k INFO ====> Epoch: 131, cost 14.83 s 2023-09-04 03:06:06,126 44k INFO ====> Epoch: 132, cost 15.27 s 2023-09-04 03:06:20,703 44k INFO ====> Epoch: 133, cost 14.58 s 2023-09-04 03:06:25,613 44k INFO Train Epoch: 134 [29%] 2023-09-04 03:06:25,613 44k INFO Losses: [2.387007713317871, 2.3182737827301025, 7.401772499084473, 18.678638458251953, 0.91362065076828], step: 2800, lr: 9.835114106370493e-05, reference_loss: 31.699312210083008 2023-09-04 03:06:35,457 44k INFO ====> Epoch: 134, cost 14.75 s 2023-09-04 03:06:50,100 44k INFO ====> Epoch: 135, cost 14.64 s 2023-09-04 03:07:04,855 44k INFO ====> Epoch: 136, cost 14.76 s 2023-09-04 03:07:19,476 44k INFO ====> Epoch: 137, cost 14.62 s 2023-09-04 03:07:33,724 44k INFO ====> Epoch: 138, cost 14.25 s 2023-09-04 03:07:48,338 44k INFO ====> Epoch: 139, cost 14.61 s 2023-09-04 03:08:03,215 44k INFO ====> Epoch: 140, cost 14.88 s 2023-09-04 03:08:17,727 44k INFO ====> Epoch: 141, cost 14.51 s 2023-09-04 03:08:32,168 44k INFO ====> Epoch: 142, cost 14.44 s 2023-09-04 03:08:44,864 44k INFO Train Epoch: 143 [81%] 2023-09-04 03:08:44,864 44k INFO Losses: [2.499281883239746, 2.2393553256988525, 5.326581001281738, 16.43933868408203, 1.0800881385803223], step: 3000, lr: 9.824055133639235e-05, reference_loss: 27.584644317626953 2023-09-04 03:08:46,965 44k INFO ====> Epoch: 143, cost 14.80 s 2023-09-04 03:09:01,521 44k INFO ====> Epoch: 144, cost 14.56 s 2023-09-04 03:09:15,997 44k INFO ====> Epoch: 145, cost 14.48 s 2023-09-04 03:09:30,434 44k INFO ====> Epoch: 146, cost 14.44 s 2023-09-04 03:09:44,849 44k INFO ====> Epoch: 147, cost 14.41 s 2023-09-04 03:09:59,521 44k INFO ====> Epoch: 148, cost 14.67 s 2023-09-04 03:10:14,132 44k INFO ====> Epoch: 149, cost 14.61 s 2023-09-04 03:10:28,483 44k INFO ====> Epoch: 150, cost 14.35 s 2023-09-04 03:10:42,864 44k INFO ====> Epoch: 151, cost 14.38 s 2023-09-04 03:10:57,455 44k INFO ====> Epoch: 152, cost 14.59 s 2023-09-04 03:11:03,112 44k INFO Train Epoch: 153 [33%] 2023-09-04 03:11:03,112 44k INFO Losses: [2.4353866577148438, 2.4125545024871826, 5.862490653991699, 16.936742782592773, 0.9401842951774597], step: 3200, lr: 9.811781969958938e-05, reference_loss: 28.587358474731445 2023-09-04 03:11:08,507 44k INFO Saving model and optimizer state at iteration 153 to ./logs\44k\G_3200.pth 2023-09-04 03:11:09,074 44k INFO Saving model and optimizer state at iteration 153 to ./logs\44k\D_3200.pth 2023-09-04 03:11:10,282 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_800.pth 2023-09-04 03:11:10,314 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_800.pth 2023-09-04 03:11:19,050 44k INFO ====> Epoch: 153, cost 21.60 s 2023-09-04 03:11:33,245 44k INFO ====> Epoch: 154, cost 14.19 s 2023-09-04 03:11:47,995 44k INFO ====> Epoch: 155, cost 14.75 s 2023-09-04 03:12:02,833 44k INFO ====> Epoch: 156, cost 14.84 s 2023-09-04 03:12:17,353 44k INFO ====> Epoch: 157, cost 14.52 s 2023-09-04 03:12:31,884 44k INFO ====> Epoch: 158, cost 14.53 s 2023-09-04 03:12:46,472 44k INFO ====> Epoch: 159, cost 14.59 s 2023-09-04 03:13:01,246 44k INFO ====> Epoch: 160, cost 14.77 s 2023-09-04 03:13:16,028 44k INFO ====> Epoch: 161, cost 14.78 s 2023-09-04 03:13:29,049 44k INFO Train Epoch: 162 [86%] 2023-09-04 03:13:29,049 44k INFO Losses: [2.2929697036743164, 2.556137800216675, 7.183849334716797, 17.497102737426758, 0.983549177646637], step: 3400, lr: 9.800749232760646e-05, reference_loss: 30.513608932495117 2023-09-04 03:13:30,410 44k INFO ====> Epoch: 162, cost 14.38 s 2023-09-04 03:13:44,884 44k INFO ====> Epoch: 163, cost 14.47 s 2023-09-04 03:13:59,696 44k INFO ====> Epoch: 164, cost 14.81 s 2023-09-04 03:14:14,253 44k INFO ====> Epoch: 165, cost 14.56 s 2023-09-04 03:14:28,651 44k INFO ====> Epoch: 166, cost 14.40 s 2023-09-04 03:14:43,024 44k INFO ====> Epoch: 167, cost 14.37 s 2023-09-04 03:14:57,769 44k INFO ====> Epoch: 168, cost 14.75 s 2023-09-04 03:15:12,578 44k INFO ====> Epoch: 169, cost 14.81 s 2023-09-04 03:15:26,950 44k INFO ====> Epoch: 170, cost 14.37 s 2023-09-04 03:15:41,364 44k INFO ====> Epoch: 171, cost 14.41 s 2023-09-04 03:15:47,708 44k INFO Train Epoch: 172 [38%] 2023-09-04 03:15:47,708 44k INFO Losses: [2.3807406425476074, 2.2620253562927246, 6.311610221862793, 17.564746856689453, 0.8789533376693726], step: 3600, lr: 9.78850518507495e-05, reference_loss: 29.3980770111084 2023-09-04 03:15:56,288 44k INFO ====> Epoch: 172, cost 14.92 s 2023-09-04 03:16:10,742 44k INFO ====> Epoch: 173, cost 14.45 s 2023-09-04 03:16:25,095 44k INFO ====> Epoch: 174, cost 14.35 s 2023-09-04 03:16:39,490 44k INFO ====> Epoch: 175, cost 14.39 s 2023-09-04 03:16:54,351 44k INFO ====> Epoch: 176, cost 14.86 s 2023-09-04 03:17:09,095 44k INFO ====> Epoch: 177, cost 14.74 s 2023-09-04 03:17:23,331 44k INFO ====> Epoch: 178, cost 14.24 s 2023-09-04 03:17:37,634 44k INFO ====> Epoch: 179, cost 14.30 s 2023-09-04 03:17:52,510 44k INFO ====> Epoch: 180, cost 14.88 s 2023-09-04 03:18:06,788 44k INFO Train Epoch: 181 [90%] 2023-09-04 03:18:06,788 44k INFO Losses: [2.361999988555908, 2.4082696437835693, 7.810259819030762, 19.43057632446289, 0.7534390091896057], step: 3800, lr: 9.777498621170277e-05, reference_loss: 32.76454544067383 2023-09-04 03:18:07,489 44k INFO ====> Epoch: 181, cost 14.98 s 2023-09-04 03:18:22,090 44k INFO ====> Epoch: 182, cost 14.60 s 2023-09-04 03:18:36,489 44k INFO ====> Epoch: 183, cost 14.40 s 2023-09-04 03:18:51,237 44k INFO ====> Epoch: 184, cost 14.75 s 2023-09-04 03:19:05,919 44k INFO ====> Epoch: 185, cost 14.68 s 2023-09-04 03:19:20,438 44k INFO ====> Epoch: 186, cost 14.52 s 2023-09-04 03:19:34,791 44k INFO ====> Epoch: 187, cost 14.35 s 2023-09-04 03:19:49,452 44k INFO ====> Epoch: 188, cost 14.66 s 2023-09-04 03:20:04,312 44k INFO ====> Epoch: 189, cost 14.86 s 2023-09-04 03:20:18,887 44k INFO ====> Epoch: 190, cost 14.58 s 2023-09-04 03:20:25,894 44k INFO Train Epoch: 191 [43%] 2023-09-04 03:20:25,895 44k INFO Losses: [2.3871052265167236, 2.3850765228271484, 6.018375873565674, 17.148923873901367, 0.9819881916046143], step: 4000, lr: 9.765283620406429e-05, reference_loss: 28.921470642089844 2023-09-04 03:20:31,239 44k INFO Saving model and optimizer state at iteration 191 to ./logs\44k\G_4000.pth 2023-09-04 03:20:31,842 44k INFO Saving model and optimizer state at iteration 191 to ./logs\44k\D_4000.pth 2023-09-04 03:20:33,096 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_1600.pth 2023-09-04 03:20:33,130 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_1600.pth 2023-09-04 03:20:40,749 44k INFO ====> Epoch: 191, cost 21.86 s 2023-09-04 03:20:55,258 44k INFO ====> Epoch: 192, cost 14.51 s 2023-09-04 03:21:09,902 44k INFO ====> Epoch: 193, cost 14.64 s 2023-09-04 03:21:24,107 44k INFO ====> Epoch: 194, cost 14.21 s 2023-09-04 03:21:38,479 44k INFO ====> Epoch: 195, cost 14.37 s 2023-09-04 03:21:53,113 44k INFO ====> Epoch: 196, cost 14.63 s 2023-09-04 03:22:07,660 44k INFO ====> Epoch: 197, cost 14.55 s 2023-09-04 03:22:22,164 44k INFO ====> Epoch: 198, cost 14.50 s 2023-09-04 03:22:36,463 44k INFO ====> Epoch: 199, cost 14.30 s 2023-09-04 03:22:51,234 44k INFO Train Epoch: 200 [95%] 2023-09-04 03:22:51,235 44k INFO Losses: [2.553147315979004, 2.144988536834717, 5.918912887573242, 17.041982650756836, 0.7911216616630554], step: 4200, lr: 9.754303167703689e-05, reference_loss: 28.450153350830078 2023-09-04 03:22:51,495 44k INFO ====> Epoch: 200, cost 15.03 s 2023-09-04 03:23:06,010 44k INFO ====> Epoch: 201, cost 14.51 s 2023-09-04 03:23:20,618 44k INFO ====> Epoch: 202, cost 14.61 s 2023-09-04 03:23:34,757 44k INFO ====> Epoch: 203, cost 14.14 s 2023-09-04 03:23:49,363 44k INFO ====> Epoch: 204, cost 14.61 s 2023-09-04 03:24:04,425 44k INFO ====> Epoch: 205, cost 15.06 s 2023-09-04 03:24:19,101 44k INFO ====> Epoch: 206, cost 14.68 s 2023-09-04 03:24:33,471 44k INFO ====> Epoch: 207, cost 14.37 s 2023-09-04 03:24:48,100 44k INFO ====> Epoch: 208, cost 14.63 s 2023-09-04 03:25:02,801 44k INFO ====> Epoch: 209, cost 14.70 s 2023-09-04 03:25:10,503 44k INFO Train Epoch: 210 [48%] 2023-09-04 03:25:10,504 44k INFO Losses: [2.5851686000823975, 2.204582929611206, 6.516208648681641, 19.32552719116211, 0.8806933164596558], step: 4400, lr: 9.742117144952805e-05, reference_loss: 31.51218032836914 2023-09-04 03:25:17,457 44k INFO ====> Epoch: 210, cost 14.66 s 2023-09-04 03:25:31,798 44k INFO ====> Epoch: 211, cost 14.34 s 2023-09-04 03:25:46,360 44k INFO ====> Epoch: 212, cost 14.56 s 2023-09-04 03:26:01,175 44k INFO ====> Epoch: 213, cost 14.82 s 2023-09-04 03:26:15,765 44k INFO ====> Epoch: 214, cost 14.59 s 2023-09-04 03:26:30,178 44k INFO ====> Epoch: 215, cost 14.41 s 2023-09-04 03:26:44,721 44k INFO ====> Epoch: 216, cost 14.54 s 2023-09-04 03:26:59,341 44k INFO ====> Epoch: 217, cost 14.62 s 2023-09-04 03:27:13,867 44k INFO ====> Epoch: 218, cost 14.53 s 2023-09-04 03:27:28,219 44k INFO ====> Epoch: 219, cost 14.35 s 2023-09-04 03:27:28,929 44k INFO Train Epoch: 220 [0%] 2023-09-04 03:27:28,930 44k INFO Losses: [2.4941463470458984, 2.3892881870269775, 6.20877742767334, 16.35572624206543, 0.9904760718345642], step: 4600, lr: 9.729946346164919e-05, reference_loss: 28.438413619995117 2023-09-04 03:27:42,850 44k INFO ====> Epoch: 220, cost 14.63 s 2023-09-04 03:27:57,349 44k INFO ====> Epoch: 221, cost 14.50 s 2023-09-04 03:28:11,872 44k INFO ====> Epoch: 222, cost 14.52 s 2023-09-04 03:28:26,357 44k INFO ====> Epoch: 223, cost 14.49 s 2023-09-04 03:28:40,713 44k INFO ====> Epoch: 224, cost 14.36 s 2023-09-04 03:28:55,298 44k INFO ====> Epoch: 225, cost 14.59 s 2023-09-04 03:29:09,939 44k INFO ====> Epoch: 226, cost 14.64 s 2023-09-04 03:29:24,425 44k INFO ====> Epoch: 227, cost 14.49 s 2023-09-04 03:29:38,818 44k INFO ====> Epoch: 228, cost 14.39 s 2023-09-04 03:29:47,530 44k INFO Train Epoch: 229 [52%] 2023-09-04 03:29:47,530 44k INFO Losses: [2.377164363861084, 2.355905055999756, 7.059666156768799, 18.924192428588867, 0.912545919418335], step: 4800, lr: 9.719005628024282e-05, reference_loss: 31.629474639892578 2023-09-04 03:29:52,870 44k INFO Saving model and optimizer state at iteration 229 to ./logs\44k\G_4800.pth 2023-09-04 03:29:53,545 44k INFO Saving model and optimizer state at iteration 229 to ./logs\44k\D_4800.pth 2023-09-04 03:29:54,681 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_2400.pth 2023-09-04 03:29:54,711 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_2400.pth 2023-09-04 03:30:00,756 44k INFO ====> Epoch: 229, cost 21.94 s 2023-09-04 03:30:15,339 44k INFO ====> Epoch: 230, cost 14.58 s 2023-09-04 03:30:29,702 44k INFO ====> Epoch: 231, cost 14.36 s 2023-09-04 03:30:44,396 44k INFO ====> Epoch: 232, cost 14.69 s 2023-09-04 03:30:59,079 44k INFO ====> Epoch: 233, cost 14.68 s 2023-09-04 03:31:13,658 44k INFO ====> Epoch: 234, cost 14.58 s 2023-09-04 03:31:28,033 44k INFO ====> Epoch: 235, cost 14.38 s 2023-09-04 03:31:42,310 44k INFO ====> Epoch: 236, cost 14.28 s 2023-09-04 03:31:57,087 44k INFO ====> Epoch: 237, cost 14.78 s 2023-09-04 03:32:11,829 44k INFO ====> Epoch: 238, cost 14.74 s 2023-09-04 03:32:13,204 44k INFO Train Epoch: 239 [5%] 2023-09-04 03:32:13,205 44k INFO Losses: [2.5339818000793457, 2.395631790161133, 5.157744884490967, 14.872054100036621, 0.8839898109436035], step: 5000, lr: 9.706863702387684e-05, reference_loss: 25.843400955200195 2023-09-04 03:32:26,591 44k INFO ====> Epoch: 239, cost 14.76 s 2023-09-04 03:32:40,871 44k INFO ====> Epoch: 240, cost 14.28 s 2023-09-04 03:32:55,521 44k INFO ====> Epoch: 241, cost 14.65 s 2023-09-04 03:33:10,038 44k INFO ====> Epoch: 242, cost 14.52 s 2023-09-04 03:33:24,336 44k INFO ====> Epoch: 243, cost 14.30 s 2023-09-04 03:33:38,718 44k INFO ====> Epoch: 244, cost 14.38 s 2023-09-04 03:33:53,754 44k INFO ====> Epoch: 245, cost 15.04 s 2023-09-04 03:34:08,579 44k INFO ====> Epoch: 246, cost 14.83 s 2023-09-04 03:34:23,135 44k INFO ====> Epoch: 247, cost 14.56 s 2023-09-04 03:34:32,178 44k INFO Train Epoch: 248 [57%] 2023-09-04 03:34:32,178 44k INFO Losses: [2.3732640743255615, 2.372363805770874, 6.755331039428711, 18.444988250732422, 0.7625980377197266], step: 5200, lr: 9.695948939241093e-05, reference_loss: 30.708545684814453 2023-09-04 03:34:37,724 44k INFO ====> Epoch: 248, cost 14.59 s 2023-09-04 03:34:52,428 44k INFO ====> Epoch: 249, cost 14.70 s 2023-09-04 03:35:06,991 44k INFO ====> Epoch: 250, cost 14.56 s 2023-09-04 03:35:21,481 44k INFO ====> Epoch: 251, cost 14.49 s 2023-09-04 03:35:35,758 44k INFO ====> Epoch: 252, cost 14.28 s 2023-09-04 03:35:50,569 44k INFO ====> Epoch: 253, cost 14.81 s 2023-09-04 03:36:05,544 44k INFO ====> Epoch: 254, cost 14.98 s 2023-09-04 03:36:20,012 44k INFO ====> Epoch: 255, cost 14.47 s 2023-09-04 03:36:34,430 44k INFO ====> Epoch: 256, cost 14.42 s 2023-09-04 03:36:48,982 44k INFO ====> Epoch: 257, cost 14.55 s 2023-09-04 03:36:51,192 44k INFO Train Epoch: 258 [10%] 2023-09-04 03:36:51,192 44k INFO Losses: [2.6578011512756348, 2.1162869930267334, 5.191831588745117, 16.913692474365234, 0.8460462689399719], step: 5400, lr: 9.683835818259144e-05, reference_loss: 27.725658416748047 2023-09-04 03:37:03,955 44k INFO ====> Epoch: 258, cost 14.97 s 2023-09-04 03:37:18,458 44k INFO ====> Epoch: 259, cost 14.50 s 2023-09-04 03:37:32,749 44k INFO ====> Epoch: 260, cost 14.29 s 2023-09-04 03:37:47,300 44k INFO ====> Epoch: 261, cost 14.55 s 2023-09-04 03:38:02,022 44k INFO ====> Epoch: 262, cost 14.72 s 2023-09-04 03:38:16,575 44k INFO ====> Epoch: 263, cost 14.55 s 2023-09-04 03:38:31,141 44k INFO ====> Epoch: 264, cost 14.57 s 2023-09-04 03:38:45,594 44k INFO ====> Epoch: 265, cost 14.45 s 2023-09-04 03:39:00,333 44k INFO ====> Epoch: 266, cost 14.74 s 2023-09-04 03:39:10,225 44k INFO Train Epoch: 267 [62%] 2023-09-04 03:39:10,225 44k INFO Losses: [2.5030956268310547, 2.3695406913757324, 5.938228130340576, 16.8659725189209, 0.8666841983795166], step: 5600, lr: 9.67294694853279e-05, reference_loss: 28.543521881103516 2023-09-04 03:39:15,547 44k INFO Saving model and optimizer state at iteration 267 to ./logs\44k\G_5600.pth 2023-09-04 03:39:16,200 44k INFO Saving model and optimizer state at iteration 267 to ./logs\44k\D_5600.pth 2023-09-04 03:39:17,862 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_3200.pth 2023-09-04 03:39:17,900 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_3200.pth 2023-09-04 03:39:22,445 44k INFO ====> Epoch: 267, cost 22.11 s 2023-09-04 03:39:36,812 44k INFO ====> Epoch: 268, cost 14.37 s 2023-09-04 03:39:51,308 44k INFO ====> Epoch: 269, cost 14.50 s 2023-09-04 03:40:05,878 44k INFO ====> Epoch: 270, cost 14.57 s 2023-09-04 03:40:20,352 44k INFO ====> Epoch: 271, cost 14.47 s 2023-09-04 03:40:34,749 44k INFO ====> Epoch: 272, cost 14.40 s 2023-09-04 03:40:49,209 44k INFO ====> Epoch: 273, cost 14.46 s 2023-09-04 03:41:04,005 44k INFO ====> Epoch: 274, cost 14.80 s 2023-09-04 03:41:18,373 44k INFO ====> Epoch: 275, cost 14.37 s 2023-09-04 03:41:32,733 44k INFO ====> Epoch: 276, cost 14.36 s 2023-09-04 03:41:35,490 44k INFO Train Epoch: 277 [14%] 2023-09-04 03:41:35,490 44k INFO Losses: [2.470017671585083, 2.3078508377075195, 7.344094753265381, 19.0073299407959, 1.0128475427627563], step: 5800, lr: 9.660862563871342e-05, reference_loss: 32.14213943481445 2023-09-04 03:41:47,714 44k INFO ====> Epoch: 277, cost 14.98 s 2023-09-04 03:42:02,415 44k INFO ====> Epoch: 278, cost 14.70 s 2023-09-04 03:42:17,005 44k INFO ====> Epoch: 279, cost 14.59 s 2023-09-04 03:42:31,446 44k INFO ====> Epoch: 280, cost 14.44 s 2023-09-04 03:42:46,110 44k INFO ====> Epoch: 281, cost 14.66 s 2023-09-04 03:43:01,087 44k INFO ====> Epoch: 282, cost 14.98 s 2023-09-04 03:43:15,474 44k INFO ====> Epoch: 283, cost 14.39 s 2023-09-04 03:43:29,736 44k INFO ====> Epoch: 284, cost 14.26 s 2023-09-04 03:43:44,346 44k INFO ====> Epoch: 285, cost 14.61 s 2023-09-04 03:43:55,026 44k INFO Train Epoch: 286 [67%] 2023-09-04 03:43:55,026 44k INFO Losses: [2.566481590270996, 2.4722230434417725, 5.995846271514893, 17.353424072265625, 0.8339502215385437], step: 6000, lr: 9.649999526137489e-05, reference_loss: 29.221925735473633 2023-09-04 03:43:59,296 44k INFO ====> Epoch: 286, cost 14.95 s 2023-09-04 03:44:13,866 44k INFO ====> Epoch: 287, cost 14.57 s 2023-09-04 03:44:28,361 44k INFO ====> Epoch: 288, cost 14.50 s 2023-09-04 03:44:42,924 44k INFO ====> Epoch: 289, cost 14.56 s 2023-09-04 03:44:57,435 44k INFO ====> Epoch: 290, cost 14.51 s 2023-09-04 03:45:12,095 44k INFO ====> Epoch: 291, cost 14.66 s 2023-09-04 03:45:26,433 44k INFO ====> Epoch: 292, cost 14.34 s 2023-09-04 03:45:40,634 44k INFO ====> Epoch: 293, cost 14.20 s 2023-09-04 03:45:55,261 44k INFO ====> Epoch: 294, cost 14.63 s 2023-09-04 03:46:09,934 44k INFO ====> Epoch: 295, cost 14.67 s 2023-09-04 03:46:13,374 44k INFO Train Epoch: 296 [19%] 2023-09-04 03:46:13,374 44k INFO Losses: [2.234588384628296, 2.613032341003418, 6.598230361938477, 16.486425399780273, 0.7535810470581055], step: 6200, lr: 9.637943809624507e-05, reference_loss: 28.68585968017578 2023-09-04 03:46:24,603 44k INFO ====> Epoch: 296, cost 14.67 s 2023-09-04 03:46:38,931 44k INFO ====> Epoch: 297, cost 14.33 s 2023-09-04 03:46:53,481 44k INFO ====> Epoch: 298, cost 14.55 s 2023-09-04 03:47:08,091 44k INFO ====> Epoch: 299, cost 14.61 s 2023-09-04 03:47:22,464 44k INFO ====> Epoch: 300, cost 14.37 s 2023-09-04 03:47:36,823 44k INFO ====> Epoch: 301, cost 14.36 s 2023-09-04 03:47:51,638 44k INFO ====> Epoch: 302, cost 14.82 s 2023-09-04 03:48:06,719 44k INFO ====> Epoch: 303, cost 15.08 s 2023-09-04 03:48:21,371 44k INFO ====> Epoch: 304, cost 14.65 s 2023-09-04 03:48:32,553 44k INFO Train Epoch: 305 [71%] 2023-09-04 03:48:32,554 44k INFO Losses: [2.5084915161132812, 2.1754977703094482, 5.955118179321289, 16.769092559814453, 0.8437416553497314], step: 6400, lr: 9.627106542601141e-05, reference_loss: 28.251941680908203 2023-09-04 03:48:37,988 44k INFO Saving model and optimizer state at iteration 305 to ./logs\44k\G_6400.pth 2023-09-04 03:48:38,670 44k INFO Saving model and optimizer state at iteration 305 to ./logs\44k\D_6400.pth 2023-09-04 03:48:39,759 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_4000.pth 2023-09-04 03:48:39,794 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_4000.pth 2023-09-04 03:48:43,068 44k INFO ====> Epoch: 305, cost 21.70 s 2023-09-04 03:48:57,892 44k INFO ====> Epoch: 306, cost 14.82 s 2023-09-04 03:49:12,645 44k INFO ====> Epoch: 307, cost 14.75 s 2023-09-04 03:49:27,052 44k INFO ====> Epoch: 308, cost 14.41 s 2023-09-04 03:49:41,464 44k INFO ====> Epoch: 309, cost 14.41 s 2023-09-04 03:49:56,172 44k INFO ====> Epoch: 310, cost 14.71 s 2023-09-04 03:50:10,811 44k INFO ====> Epoch: 311, cost 14.64 s 2023-09-04 03:50:25,139 44k INFO ====> Epoch: 312, cost 14.33 s 2023-09-04 03:50:39,571 44k INFO ====> Epoch: 313, cost 14.43 s 2023-09-04 03:50:54,305 44k INFO ====> Epoch: 314, cost 14.73 s 2023-09-04 03:50:58,571 44k INFO Train Epoch: 315 [24%] 2023-09-04 03:50:58,572 44k INFO Losses: [2.527101993560791, 2.3678267002105713, 5.905847549438477, 16.879018783569336, 0.8097784519195557], step: 6600, lr: 9.615079426226314e-05, reference_loss: 28.489572525024414 2023-09-04 03:51:09,282 44k INFO ====> Epoch: 315, cost 14.98 s 2023-09-04 03:51:23,415 44k INFO ====> Epoch: 316, cost 14.13 s 2023-09-04 03:51:37,597 44k INFO ====> Epoch: 317, cost 14.18 s 2023-09-04 03:51:52,060 44k INFO ====> Epoch: 318, cost 14.46 s 2023-09-04 03:52:06,655 44k INFO ====> Epoch: 319, cost 14.59 s 2023-09-04 03:52:21,236 44k INFO ====> Epoch: 320, cost 14.58 s 2023-09-04 03:52:35,639 44k INFO ====> Epoch: 321, cost 14.40 s 2023-09-04 03:52:50,009 44k INFO ====> Epoch: 322, cost 14.37 s 2023-09-04 03:53:04,644 44k INFO ====> Epoch: 323, cost 14.63 s 2023-09-04 03:53:16,467 44k INFO Train Epoch: 324 [76%] 2023-09-04 03:53:16,467 44k INFO Losses: [2.346024513244629, 2.549798011779785, 7.298701286315918, 19.619401931762695, 0.7979968786239624], step: 6800, lr: 9.604267868776807e-05, reference_loss: 32.61192321777344 2023-09-04 03:53:19,200 44k INFO ====> Epoch: 324, cost 14.56 s 2023-09-04 03:53:33,647 44k INFO ====> Epoch: 325, cost 14.45 s 2023-09-04 03:53:48,495 44k INFO ====> Epoch: 326, cost 14.85 s 2023-09-04 03:54:03,481 44k INFO ====> Epoch: 327, cost 14.99 s 2023-09-04 03:54:18,155 44k INFO ====> Epoch: 328, cost 14.67 s 2023-09-04 03:54:32,634 44k INFO ====> Epoch: 329, cost 14.48 s 2023-09-04 03:54:47,373 44k INFO ====> Epoch: 330, cost 14.74 s 2023-09-04 03:55:02,125 44k INFO ====> Epoch: 331, cost 14.75 s 2023-09-04 03:55:16,584 44k INFO ====> Epoch: 332, cost 14.46 s 2023-09-04 03:55:30,951 44k INFO ====> Epoch: 333, cost 14.37 s 2023-09-04 03:55:35,833 44k INFO Train Epoch: 334 [29%] 2023-09-04 03:55:35,834 44k INFO Losses: [2.4200944900512695, 2.279477834701538, 6.978575706481934, 15.966625213623047, 0.7250052690505981], step: 7000, lr: 9.592269284691169e-05, reference_loss: 28.369779586791992 2023-09-04 03:55:45,827 44k INFO ====> Epoch: 334, cost 14.88 s 2023-09-04 03:56:00,433 44k INFO ====> Epoch: 335, cost 14.61 s 2023-09-04 03:56:15,013 44k INFO ====> Epoch: 336, cost 14.58 s 2023-09-04 03:56:29,579 44k INFO ====> Epoch: 337, cost 14.57 s 2023-09-04 03:56:43,946 44k INFO ====> Epoch: 338, cost 14.37 s 2023-09-04 03:56:58,508 44k INFO ====> Epoch: 339, cost 14.56 s 2023-09-04 03:57:13,122 44k INFO ====> Epoch: 340, cost 14.61 s 2023-09-04 03:57:27,371 44k INFO ====> Epoch: 341, cost 14.25 s 2023-09-04 03:57:41,623 44k INFO ====> Epoch: 342, cost 14.25 s 2023-09-04 03:57:54,397 44k INFO Train Epoch: 343 [81%] 2023-09-04 03:57:54,398 44k INFO Losses: [2.4048337936401367, 2.4485669136047363, 6.3834452629089355, 16.505224227905273, 0.8378720283508301], step: 7200, lr: 9.581483375823925e-05, reference_loss: 28.579940795898438 2023-09-04 03:57:59,766 44k INFO Saving model and optimizer state at iteration 343 to ./logs\44k\G_7200.pth 2023-09-04 03:58:00,404 44k INFO Saving model and optimizer state at iteration 343 to ./logs\44k\D_7200.pth 2023-09-04 03:58:02,116 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_4800.pth 2023-09-04 03:58:02,146 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_4800.pth 2023-09-04 03:58:03,951 44k INFO ====> Epoch: 343, cost 22.33 s 2023-09-04 03:58:18,303 44k INFO ====> Epoch: 344, cost 14.35 s 2023-09-04 03:58:32,746 44k INFO ====> Epoch: 345, cost 14.44 s 2023-09-04 03:58:47,486 44k INFO ====> Epoch: 346, cost 14.74 s 2023-09-04 03:59:02,173 44k INFO ====> Epoch: 347, cost 14.69 s 2023-09-04 03:59:16,779 44k INFO ====> Epoch: 348, cost 14.61 s 2023-09-04 03:59:31,100 44k INFO ====> Epoch: 349, cost 14.32 s 2023-09-04 03:59:45,578 44k INFO ====> Epoch: 350, cost 14.48 s 2023-09-04 04:00:00,410 44k INFO ====> Epoch: 351, cost 14.83 s 2023-09-04 04:00:15,524 44k INFO ====> Epoch: 352, cost 15.11 s 2023-09-04 04:00:21,106 44k INFO Train Epoch: 353 [33%] 2023-09-04 04:00:21,106 44k INFO Losses: [2.50160551071167, 2.389134645462036, 5.925080299377441, 15.55258560180664, 0.8810001611709595], step: 7400, lr: 9.569513256339471e-05, reference_loss: 27.249406814575195 2023-09-04 04:00:30,371 44k INFO ====> Epoch: 353, cost 14.85 s 2023-09-04 04:00:44,978 44k INFO ====> Epoch: 354, cost 14.61 s 2023-09-04 04:00:59,762 44k INFO ====> Epoch: 355, cost 14.78 s 2023-09-04 04:01:14,239 44k INFO ====> Epoch: 356, cost 14.48 s 2023-09-04 04:01:28,584 44k INFO ====> Epoch: 357, cost 14.35 s 2023-09-04 04:01:42,867 44k INFO ====> Epoch: 358, cost 14.28 s 2023-09-04 04:01:58,209 44k INFO ====> Epoch: 359, cost 15.34 s 2023-09-04 04:02:13,124 44k INFO ====> Epoch: 360, cost 14.92 s 2023-09-04 04:02:27,727 44k INFO ====> Epoch: 361, cost 14.60 s 2023-09-04 04:02:41,124 44k INFO Train Epoch: 362 [86%] 2023-09-04 04:02:41,124 44k INFO Losses: [2.3799526691436768, 2.4261555671691895, 6.026432991027832, 15.65579891204834, 0.7425792813301086], step: 7600, lr: 9.558752935207586e-05, reference_loss: 27.230918884277344 2023-09-04 04:02:42,552 44k INFO ====> Epoch: 362, cost 14.82 s 2023-09-04 04:02:57,112 44k INFO ====> Epoch: 363, cost 14.56 s 2023-09-04 04:03:11,544 44k INFO ====> Epoch: 364, cost 14.43 s 2023-09-04 04:03:26,136 44k INFO ====> Epoch: 365, cost 14.59 s 2023-09-04 04:03:40,576 44k INFO ====> Epoch: 366, cost 14.44 s 2023-09-04 04:03:55,282 44k INFO ====> Epoch: 367, cost 14.71 s 2023-09-04 04:04:10,112 44k INFO ====> Epoch: 368, cost 14.83 s 2023-09-04 04:04:24,648 44k INFO ====> Epoch: 369, cost 14.54 s 2023-09-04 04:04:39,099 44k INFO ====> Epoch: 370, cost 14.45 s 2023-09-04 04:04:53,794 44k INFO ====> Epoch: 371, cost 14.70 s 2023-09-04 04:05:00,305 44k INFO Train Epoch: 372 [38%] 2023-09-04 04:05:00,305 44k INFO Losses: [2.4534637928009033, 2.3384389877319336, 6.2957000732421875, 17.315465927124023, 0.6862635612487793], step: 7800, lr: 9.546811212796888e-05, reference_loss: 29.089332580566406 2023-09-04 04:05:08,835 44k INFO ====> Epoch: 372, cost 15.04 s 2023-09-04 04:05:23,318 44k INFO ====> Epoch: 373, cost 14.48 s 2023-09-04 04:05:37,790 44k INFO ====> Epoch: 374, cost 14.47 s 2023-09-04 04:05:52,740 44k INFO ====> Epoch: 375, cost 14.95 s 2023-09-04 04:06:07,639 44k INFO ====> Epoch: 376, cost 14.90 s 2023-09-04 04:06:22,214 44k INFO ====> Epoch: 377, cost 14.57 s 2023-09-04 04:06:36,571 44k INFO ====> Epoch: 378, cost 14.36 s 2023-09-04 04:06:51,037 44k INFO ====> Epoch: 379, cost 14.47 s 2023-09-04 04:07:05,415 44k INFO ====> Epoch: 380, cost 14.38 s 2023-09-04 04:07:19,657 44k INFO Train Epoch: 381 [90%] 2023-09-04 04:07:19,657 44k INFO Losses: [2.4995501041412354, 2.2754509449005127, 6.2464680671691895, 16.584095001220703, 0.915495753288269], step: 8000, lr: 9.536076418697815e-05, reference_loss: 28.521059036254883 2023-09-04 04:07:25,079 44k INFO Saving model and optimizer state at iteration 381 to ./logs\44k\G_8000.pth 2023-09-04 04:07:25,709 44k INFO Saving model and optimizer state at iteration 381 to ./logs\44k\D_8000.pth 2023-09-04 04:07:26,871 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_5600.pth 2023-09-04 04:07:26,907 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_5600.pth 2023-09-04 04:07:27,399 44k INFO ====> Epoch: 381, cost 21.98 s 2023-09-04 04:07:42,173 44k INFO ====> Epoch: 382, cost 14.77 s 2023-09-04 04:07:56,861 44k INFO ====> Epoch: 383, cost 14.69 s 2023-09-04 04:08:11,395 44k INFO ====> Epoch: 384, cost 14.53 s 2023-09-04 04:08:26,279 44k INFO ====> Epoch: 385, cost 14.88 s 2023-09-04 04:08:40,827 44k INFO ====> Epoch: 386, cost 14.55 s 2023-09-04 04:08:55,243 44k INFO ====> Epoch: 387, cost 14.42 s 2023-09-04 04:09:09,643 44k INFO ====> Epoch: 388, cost 14.40 s 2023-09-04 04:09:24,031 44k INFO ====> Epoch: 389, cost 14.39 s 2023-09-04 04:09:38,517 44k INFO ====> Epoch: 390, cost 14.49 s 2023-09-04 04:09:45,709 44k INFO Train Epoch: 391 [43%] 2023-09-04 04:09:45,709 44k INFO Losses: [2.3506133556365967, 2.5596697330474854, 7.198373317718506, 17.64354705810547, 0.779762864112854], step: 8200, lr: 9.524163025993642e-05, reference_loss: 30.531965255737305 2023-09-04 04:09:53,432 44k INFO ====> Epoch: 391, cost 14.91 s 2023-09-04 04:10:07,896 44k INFO ====> Epoch: 392, cost 14.46 s 2023-09-04 04:10:22,514 44k INFO ====> Epoch: 393, cost 14.62 s 2023-09-04 04:10:36,945 44k INFO ====> Epoch: 394, cost 14.43 s 2023-09-04 04:10:51,684 44k INFO ====> Epoch: 395, cost 14.74 s 2023-09-04 04:11:06,226 44k INFO ====> Epoch: 396, cost 14.54 s 2023-09-04 04:11:20,787 44k INFO ====> Epoch: 397, cost 14.56 s 2023-09-04 04:11:35,215 44k INFO ====> Epoch: 398, cost 14.43 s 2023-09-04 04:11:49,798 44k INFO ====> Epoch: 399, cost 14.58 s 2023-09-04 04:12:04,692 44k INFO Train Epoch: 400 [95%] 2023-09-04 04:12:04,692 44k INFO Losses: [2.3765833377838135, 2.5342228412628174, 8.950823783874512, 18.353134155273438, 0.4853152930736542], step: 8400, lr: 9.513453698368834e-05, reference_loss: 32.700077056884766 2023-09-04 04:12:04,951 44k INFO ====> Epoch: 400, cost 15.15 s 2023-09-04 04:12:19,453 44k INFO ====> Epoch: 401, cost 14.50 s 2023-09-04 04:12:33,929 44k INFO ====> Epoch: 402, cost 14.48 s 2023-09-04 04:12:48,697 44k INFO ====> Epoch: 403, cost 14.77 s 2023-09-04 04:13:03,513 44k INFO ====> Epoch: 404, cost 14.82 s 2023-09-04 04:13:18,038 44k INFO ====> Epoch: 405, cost 14.52 s 2023-09-04 04:13:32,217 44k INFO ====> Epoch: 406, cost 14.18 s 2023-09-04 04:13:46,834 44k INFO ====> Epoch: 407, cost 14.62 s 2023-09-04 04:14:01,605 44k INFO ====> Epoch: 408, cost 14.77 s 2023-09-04 04:14:16,132 44k INFO ====> Epoch: 409, cost 14.53 s 2023-09-04 04:14:23,879 44k INFO Train Epoch: 410 [48%] 2023-09-04 04:14:23,880 44k INFO Losses: [2.4310154914855957, 2.485517978668213, 6.836060047149658, 17.652990341186523, 0.7668866515159607], step: 8600, lr: 9.501568568163774e-05, reference_loss: 30.172470092773438 2023-09-04 04:14:30,940 44k INFO ====> Epoch: 410, cost 14.81 s 2023-09-04 04:14:45,410 44k INFO ====> Epoch: 411, cost 14.47 s 2023-09-04 04:14:59,915 44k INFO ====> Epoch: 412, cost 14.50 s 2023-09-04 04:15:14,393 44k INFO ====> Epoch: 413, cost 14.48 s 2023-09-04 04:15:28,527 44k INFO ====> Epoch: 414, cost 14.13 s 2023-09-04 04:15:42,781 44k INFO ====> Epoch: 415, cost 14.25 s 2023-09-04 04:15:57,480 44k INFO ====> Epoch: 416, cost 14.70 s 2023-09-04 04:16:12,108 44k INFO ====> Epoch: 417, cost 14.63 s 2023-09-04 04:16:26,387 44k INFO ====> Epoch: 418, cost 14.28 s 2023-09-04 04:16:40,761 44k INFO ====> Epoch: 419, cost 14.37 s 2023-09-04 04:16:41,476 44k INFO Train Epoch: 420 [0%] 2023-09-04 04:16:41,476 44k INFO Losses: [2.2666988372802734, 2.4993555545806885, 6.569221019744873, 17.080801010131836, 0.7954935431480408], step: 8800, lr: 9.489698286017521e-05, reference_loss: 29.211570739746094 2023-09-04 04:16:47,181 44k INFO Saving model and optimizer state at iteration 420 to ./logs\44k\G_8800.pth 2023-09-04 04:16:47,772 44k INFO Saving model and optimizer state at iteration 420 to ./logs\44k\D_8800.pth 2023-09-04 04:16:49,033 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_6400.pth 2023-09-04 04:16:49,067 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_6400.pth 2023-09-04 04:17:03,059 44k INFO ====> Epoch: 420, cost 22.30 s 2023-09-04 04:17:17,540 44k INFO ====> Epoch: 421, cost 14.48 s 2023-09-04 04:17:31,819 44k INFO ====> Epoch: 422, cost 14.28 s 2023-09-04 04:17:46,479 44k INFO ====> Epoch: 423, cost 14.66 s 2023-09-04 04:18:01,206 44k INFO ====> Epoch: 424, cost 14.73 s 2023-09-04 04:18:16,041 44k INFO ====> Epoch: 425, cost 14.83 s 2023-09-04 04:18:30,614 44k INFO ====> Epoch: 426, cost 14.57 s 2023-09-04 04:18:45,373 44k INFO ====> Epoch: 427, cost 14.76 s 2023-09-04 04:19:00,316 44k INFO ====> Epoch: 428, cost 14.94 s 2023-09-04 04:19:08,953 44k INFO Train Epoch: 429 [52%] 2023-09-04 04:19:08,953 44k INFO Losses: [2.339188575744629, 2.5246081352233887, 6.632565975189209, 17.7928409576416, 0.8504566550254822], step: 9000, lr: 9.479027711844423e-05, reference_loss: 30.139659881591797 2023-09-04 04:19:15,163 44k INFO ====> Epoch: 429, cost 14.85 s 2023-09-04 04:19:29,531 44k INFO ====> Epoch: 430, cost 14.37 s 2023-09-04 04:19:44,244 44k INFO ====> Epoch: 431, cost 14.71 s 2023-09-04 04:19:59,043 44k INFO ====> Epoch: 432, cost 14.80 s 2023-09-04 04:20:13,612 44k INFO ====> Epoch: 433, cost 14.57 s 2023-09-04 04:20:28,022 44k INFO ====> Epoch: 434, cost 14.41 s 2023-09-04 04:20:42,407 44k INFO ====> Epoch: 435, cost 14.38 s 2023-09-04 04:20:56,995 44k INFO ====> Epoch: 436, cost 14.59 s 2023-09-04 04:21:11,586 44k INFO ====> Epoch: 437, cost 14.59 s 2023-09-04 04:21:25,834 44k INFO ====> Epoch: 438, cost 14.25 s 2023-09-04 04:21:27,210 44k INFO Train Epoch: 439 [5%] 2023-09-04 04:21:27,210 44k INFO Losses: [2.1708805561065674, 2.6769769191741943, 8.037736892700195, 17.20783805847168, 0.7233844995498657], step: 9200, lr: 9.467185589924815e-05, reference_loss: 30.816818237304688 2023-09-04 04:21:40,296 44k INFO ====> Epoch: 439, cost 14.46 s 2023-09-04 04:21:54,703 44k INFO ====> Epoch: 440, cost 14.41 s 2023-09-04 04:22:09,521 44k INFO ====> Epoch: 441, cost 14.82 s 2023-09-04 04:22:23,932 44k INFO ====> Epoch: 442, cost 14.41 s 2023-09-04 04:22:38,336 44k INFO ====> Epoch: 443, cost 14.40 s 2023-09-04 04:22:53,094 44k INFO ====> Epoch: 444, cost 14.76 s 2023-09-04 04:23:07,965 44k INFO ====> Epoch: 445, cost 14.87 s 2023-09-04 04:23:22,320 44k INFO ====> Epoch: 446, cost 14.35 s 2023-09-04 04:23:36,558 44k INFO ====> Epoch: 447, cost 14.24 s 2023-09-04 04:23:45,783 44k INFO Train Epoch: 448 [57%] 2023-09-04 04:23:45,783 44k INFO Losses: [2.2600231170654297, 2.653318166732788, 7.287391662597656, 17.89423370361328, 0.6535707712173462], step: 9400, lr: 9.456540329875122e-05, reference_loss: 30.748537063598633 2023-09-04 04:23:51,475 44k INFO ====> Epoch: 448, cost 14.92 s 2023-09-04 04:24:06,308 44k INFO ====> Epoch: 449, cost 14.83 s 2023-09-04 04:24:20,950 44k INFO ====> Epoch: 450, cost 14.64 s 2023-09-04 04:24:35,725 44k INFO ====> Epoch: 451, cost 14.77 s 2023-09-04 04:24:50,310 44k INFO ====> Epoch: 452, cost 14.59 s 2023-09-04 04:25:05,133 44k INFO ====> Epoch: 453, cost 14.82 s 2023-09-04 04:25:19,536 44k INFO ====> Epoch: 454, cost 14.40 s 2023-09-04 04:25:33,577 44k INFO ====> Epoch: 455, cost 14.04 s 2023-09-04 04:25:48,200 44k INFO ====> Epoch: 456, cost 14.62 s 2023-09-04 04:26:03,060 44k INFO ====> Epoch: 457, cost 14.86 s 2023-09-04 04:26:05,255 44k INFO Train Epoch: 458 [10%] 2023-09-04 04:26:05,255 44k INFO Losses: [2.547600507736206, 2.2764132022857666, 5.983936309814453, 16.707578659057617, 0.8720351457595825], step: 9600, lr: 9.4447263013768e-05, reference_loss: 28.38756561279297 2023-09-04 04:26:10,565 44k INFO Saving model and optimizer state at iteration 458 to ./logs\44k\G_9600.pth 2023-09-04 04:26:11,122 44k INFO Saving model and optimizer state at iteration 458 to ./logs\44k\D_9600.pth 2023-09-04 04:26:12,444 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_7200.pth 2023-09-04 04:26:12,481 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_7200.pth 2023-09-04 04:26:24,901 44k INFO ====> Epoch: 458, cost 21.84 s 2023-09-04 04:26:39,299 44k INFO ====> Epoch: 459, cost 14.40 s 2023-09-04 04:26:53,973 44k INFO ====> Epoch: 460, cost 14.67 s 2023-09-04 04:27:08,455 44k INFO ====> Epoch: 461, cost 14.48 s 2023-09-04 04:27:22,569 44k INFO ====> Epoch: 462, cost 14.11 s 2023-09-04 04:27:36,776 44k INFO ====> Epoch: 463, cost 14.21 s 2023-09-04 04:27:51,394 44k INFO ====> Epoch: 464, cost 14.62 s 2023-09-04 04:28:06,038 44k INFO ====> Epoch: 465, cost 14.64 s 2023-09-04 04:28:20,488 44k INFO ====> Epoch: 466, cost 14.45 s 2023-09-04 04:28:30,169 44k INFO Train Epoch: 467 [62%] 2023-09-04 04:28:30,169 44k INFO Losses: [2.415132522583008, 2.400944948196411, 6.908298969268799, 17.969453811645508, 0.6542684435844421], step: 9800, lr: 9.434106295397058e-05, reference_loss: 30.348098754882812 2023-09-04 04:28:34,978 44k INFO ====> Epoch: 467, cost 14.49 s 2023-09-04 04:28:49,573 44k INFO ====> Epoch: 468, cost 14.59 s 2023-09-04 04:29:04,290 44k INFO ====> Epoch: 469, cost 14.72 s 2023-09-04 04:29:18,696 44k INFO ====> Epoch: 470, cost 14.41 s 2023-09-04 04:29:32,964 44k INFO ====> Epoch: 471, cost 14.27 s 2023-09-04 04:29:47,675 44k INFO ====> Epoch: 472, cost 14.71 s 2023-09-04 04:30:02,550 44k INFO ====> Epoch: 473, cost 14.87 s 2023-09-04 04:30:17,220 44k INFO ====> Epoch: 474, cost 14.67 s 2023-09-04 04:30:31,680 44k INFO ====> Epoch: 475, cost 14.46 s 2023-09-04 04:30:46,154 44k INFO ====> Epoch: 476, cost 14.47 s 2023-09-04 04:30:48,979 44k INFO Train Epoch: 477 [14%] 2023-09-04 04:30:48,980 44k INFO Losses: [2.569040536880493, 1.9650083780288696, 6.050761699676514, 17.481470108032227, 0.7041351199150085], step: 10000, lr: 9.422320293673162e-05, reference_loss: 28.770416259765625 2023-09-04 04:31:01,237 44k INFO ====> Epoch: 477, cost 15.08 s 2023-09-04 04:31:15,694 44k INFO ====> Epoch: 478, cost 14.46 s 2023-09-04 04:31:29,902 44k INFO ====> Epoch: 479, cost 14.21 s 2023-09-04 04:31:44,438 44k INFO ====> Epoch: 480, cost 14.54 s 2023-09-04 04:31:59,175 44k INFO ====> Epoch: 481, cost 14.74 s 2023-09-04 04:32:14,075 44k INFO ====> Epoch: 482, cost 14.90 s 2023-09-04 04:32:28,602 44k INFO ====> Epoch: 483, cost 14.53 s 2023-09-04 04:32:43,071 44k INFO ====> Epoch: 484, cost 14.47 s 2023-09-04 04:32:57,662 44k INFO ====> Epoch: 485, cost 14.59 s 2023-09-04 04:33:08,349 44k INFO Train Epoch: 486 [67%] 2023-09-04 04:33:08,350 44k INFO Losses: [2.07772159576416, 2.5745487213134766, 6.550475597381592, 14.742013931274414, 0.6986162066459656], step: 10200, lr: 9.411725481852385e-05, reference_loss: 26.643375396728516 2023-09-04 04:33:12,488 44k INFO ====> Epoch: 486, cost 14.83 s 2023-09-04 04:33:26,827 44k INFO ====> Epoch: 487, cost 14.34 s 2023-09-04 04:33:41,126 44k INFO ====> Epoch: 488, cost 14.30 s 2023-09-04 04:33:55,525 44k INFO ====> Epoch: 489, cost 14.40 s 2023-09-04 04:34:10,042 44k INFO ====> Epoch: 490, cost 14.52 s 2023-09-04 04:34:24,223 44k INFO ====> Epoch: 491, cost 14.18 s 2023-09-04 04:34:38,587 44k INFO ====> Epoch: 492, cost 14.36 s 2023-09-04 04:34:53,187 44k INFO ====> Epoch: 493, cost 14.60 s 2023-09-04 04:35:07,736 44k INFO ====> Epoch: 494, cost 14.55 s 2023-09-04 04:35:21,888 44k INFO ====> Epoch: 495, cost 14.15 s 2023-09-04 04:35:25,386 44k INFO Train Epoch: 496 [19%] 2023-09-04 04:35:25,386 44k INFO Losses: [2.530318260192871, 2.419818878173828, 6.614101409912109, 17.316091537475586, 0.8166894912719727], step: 10400, lr: 9.399967440414155e-05, reference_loss: 29.697017669677734 2023-09-04 04:35:30,751 44k INFO Saving model and optimizer state at iteration 496 to ./logs\44k\G_10400.pth 2023-09-04 04:35:31,356 44k INFO Saving model and optimizer state at iteration 496 to ./logs\44k\D_10400.pth 2023-09-04 04:35:32,512 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_8000.pth 2023-09-04 04:35:32,550 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_8000.pth 2023-09-04 04:35:43,524 44k INFO ====> Epoch: 496, cost 21.64 s 2023-09-04 04:35:58,265 44k INFO ====> Epoch: 497, cost 14.74 s 2023-09-04 04:36:13,165 44k INFO ====> Epoch: 498, cost 14.90 s 2023-09-04 04:36:27,579 44k INFO ====> Epoch: 499, cost 14.41 s 2023-09-04 04:36:42,163 44k INFO ====> Epoch: 500, cost 14.58 s 2023-09-04 04:36:56,970 44k INFO ====> Epoch: 501, cost 14.81 s 2023-09-04 04:37:11,636 44k INFO ====> Epoch: 502, cost 14.67 s 2023-09-04 04:37:25,842 44k INFO ====> Epoch: 503, cost 14.21 s 2023-09-04 04:37:40,187 44k INFO ====> Epoch: 504, cost 14.34 s 2023-09-04 04:37:51,667 44k INFO Train Epoch: 505 [71%] 2023-09-04 04:37:51,667 44k INFO Losses: [2.351020336151123, 2.5181422233581543, 6.597741603851318, 15.867526054382324, 0.9164776802062988], step: 10600, lr: 9.389397762983476e-05, reference_loss: 28.25090789794922 2023-09-04 04:37:55,280 44k INFO ====> Epoch: 505, cost 15.09 s 2023-09-04 04:38:09,941 44k INFO ====> Epoch: 506, cost 14.66 s 2023-09-04 04:38:24,264 44k INFO ====> Epoch: 507, cost 14.32 s 2023-09-04 04:38:38,672 44k INFO ====> Epoch: 508, cost 14.41 s 2023-09-04 04:38:53,268 44k INFO ====> Epoch: 509, cost 14.60 s 2023-09-04 04:39:07,982 44k INFO ====> Epoch: 510, cost 14.71 s 2023-09-04 04:39:22,294 44k INFO ====> Epoch: 511, cost 14.31 s 2023-09-04 04:39:36,645 44k INFO ====> Epoch: 512, cost 14.35 s 2023-09-04 04:39:51,095 44k INFO ====> Epoch: 513, cost 14.45 s 2023-09-04 04:40:05,732 44k INFO ====> Epoch: 514, cost 14.64 s 2023-09-04 04:40:09,979 44k INFO Train Epoch: 515 [24%] 2023-09-04 04:40:09,980 44k INFO Losses: [2.529862880706787, 2.3114094734191895, 6.504950523376465, 17.16285514831543, 0.6276082396507263], step: 10800, lr: 9.377667615499888e-05, reference_loss: 29.136686325073242 2023-09-04 04:40:20,396 44k INFO ====> Epoch: 515, cost 14.66 s 2023-09-04 04:40:34,696 44k INFO ====> Epoch: 516, cost 14.30 s 2023-09-04 04:40:49,456 44k INFO ====> Epoch: 517, cost 14.76 s 2023-09-04 04:41:04,275 44k INFO ====> Epoch: 518, cost 14.82 s 2023-09-04 04:41:18,786 44k INFO ====> Epoch: 519, cost 14.51 s 2023-09-04 04:41:33,052 44k INFO ====> Epoch: 520, cost 14.27 s 2023-09-04 04:41:47,781 44k INFO ====> Epoch: 521, cost 14.73 s 2023-09-04 04:42:02,629 44k INFO ====> Epoch: 522, cost 14.85 s 2023-09-04 04:42:17,152 44k INFO ====> Epoch: 523, cost 14.52 s 2023-09-04 04:42:29,251 44k INFO Train Epoch: 524 [76%] 2023-09-04 04:42:29,251 44k INFO Losses: [2.5277304649353027, 2.14292049407959, 5.26121711730957, 14.659408569335938, 0.6372537016868591], step: 11000, lr: 9.367123012832248e-05, reference_loss: 25.22852897644043 2023-09-04 04:42:32,045 44k INFO ====> Epoch: 524, cost 14.89 s 2023-09-04 04:42:46,572 44k INFO ====> Epoch: 525, cost 14.53 s 2023-09-04 04:43:01,285 44k INFO ====> Epoch: 526, cost 14.71 s 2023-09-04 04:43:15,914 44k INFO ====> Epoch: 527, cost 14.63 s 2023-09-04 04:43:30,217 44k INFO ====> Epoch: 528, cost 14.30 s 2023-09-04 04:43:44,576 44k INFO ====> Epoch: 529, cost 14.36 s 2023-09-04 04:43:59,206 44k INFO ====> Epoch: 530, cost 14.63 s 2023-09-04 04:44:13,908 44k INFO ====> Epoch: 531, cost 14.70 s 2023-09-04 04:44:28,231 44k INFO ====> Epoch: 532, cost 14.32 s 2023-09-04 04:44:42,666 44k INFO ====> Epoch: 533, cost 14.43 s 2023-09-04 04:44:47,626 44k INFO Train Epoch: 534 [29%] 2023-09-04 04:44:47,626 44k INFO Losses: [2.4671316146850586, 2.4285240173339844, 7.817749500274658, 18.830257415771484, 0.9035239219665527], step: 11200, lr: 9.355420693129632e-05, reference_loss: 32.44718551635742 2023-09-04 04:44:52,947 44k INFO Saving model and optimizer state at iteration 534 to ./logs\44k\G_11200.pth 2023-09-04 04:44:53,501 44k INFO Saving model and optimizer state at iteration 534 to ./logs\44k\D_11200.pth 2023-09-04 04:44:54,703 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_8800.pth 2023-09-04 04:44:54,744 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_8800.pth 2023-09-04 04:45:04,477 44k INFO ====> Epoch: 534, cost 21.81 s 2023-09-04 04:45:18,914 44k INFO ====> Epoch: 535, cost 14.44 s 2023-09-04 04:45:33,251 44k INFO ====> Epoch: 536, cost 14.34 s 2023-09-04 04:45:47,666 44k INFO ====> Epoch: 537, cost 14.41 s 2023-09-04 04:46:02,236 44k INFO ====> Epoch: 538, cost 14.57 s 2023-09-04 04:46:16,746 44k INFO ====> Epoch: 539, cost 14.51 s 2023-09-04 04:46:31,202 44k INFO ====> Epoch: 540, cost 14.46 s 2023-09-04 04:46:45,798 44k INFO ====> Epoch: 541, cost 14.60 s 2023-09-04 04:47:00,531 44k INFO ====> Epoch: 542, cost 14.73 s 2023-09-04 04:47:13,244 44k INFO Train Epoch: 543 [81%] 2023-09-04 04:47:13,245 44k INFO Losses: [2.3507468700408936, 2.314894676208496, 6.790603160858154, 17.747018814086914, 0.8166838884353638], step: 11400, lr: 9.344901105739411e-05, reference_loss: 30.019948959350586 2023-09-04 04:47:15,347 44k INFO ====> Epoch: 543, cost 14.82 s 2023-09-04 04:47:29,598 44k INFO ====> Epoch: 544, cost 14.25 s 2023-09-04 04:47:44,026 44k INFO ====> Epoch: 545, cost 14.43 s 2023-09-04 04:47:58,775 44k INFO ====> Epoch: 546, cost 14.75 s 2023-09-04 04:48:13,611 44k INFO ====> Epoch: 547, cost 14.84 s 2023-09-04 04:48:28,017 44k INFO ====> Epoch: 548, cost 14.41 s 2023-09-04 04:48:42,480 44k INFO ====> Epoch: 549, cost 14.46 s 2023-09-04 04:48:56,991 44k INFO ====> Epoch: 550, cost 14.51 s 2023-09-04 04:49:11,757 44k INFO ====> Epoch: 551, cost 14.77 s 2023-09-04 04:49:25,884 44k INFO ====> Epoch: 552, cost 14.13 s 2023-09-04 04:49:31,359 44k INFO Train Epoch: 553 [33%] 2023-09-04 04:49:31,359 44k INFO Losses: [2.424280881881714, 2.300621509552002, 6.8760151863098145, 17.430572509765625, 0.8038735389709473], step: 11600, lr: 9.33322654780109e-05, reference_loss: 29.835363388061523 2023-09-04 04:49:40,407 44k INFO ====> Epoch: 553, cost 14.52 s 2023-09-04 04:49:55,162 44k INFO ====> Epoch: 554, cost 14.76 s 2023-09-04 04:50:09,760 44k INFO ====> Epoch: 555, cost 14.60 s 2023-09-04 04:50:24,128 44k INFO ====> Epoch: 556, cost 14.37 s 2023-09-04 04:50:38,625 44k INFO ====> Epoch: 557, cost 14.50 s 2023-09-04 04:50:53,158 44k INFO ====> Epoch: 558, cost 14.53 s 2023-09-04 04:51:07,825 44k INFO ====> Epoch: 559, cost 14.67 s 2023-09-04 04:51:22,128 44k INFO ====> Epoch: 560, cost 14.30 s 2023-09-04 04:51:36,399 44k INFO ====> Epoch: 561, cost 14.27 s 2023-09-04 04:51:49,754 44k INFO Train Epoch: 562 [86%] 2023-09-04 04:51:49,754 44k INFO Losses: [2.314772129058838, 3.0127692222595215, 7.315591812133789, 16.20864486694336, 0.8129470944404602], step: 11800, lr: 9.322731916343797e-05, reference_loss: 29.664724349975586 2023-09-04 04:51:51,159 44k INFO ====> Epoch: 562, cost 14.76 s 2023-09-04 04:52:05,828 44k INFO ====> Epoch: 563, cost 14.67 s 2023-09-04 04:52:20,128 44k INFO ====> Epoch: 564, cost 14.30 s 2023-09-04 04:52:34,518 44k INFO ====> Epoch: 565, cost 14.39 s 2023-09-04 04:52:48,986 44k INFO ====> Epoch: 566, cost 14.47 s 2023-09-04 04:53:03,524 44k INFO ====> Epoch: 567, cost 14.54 s 2023-09-04 04:53:18,044 44k INFO ====> Epoch: 568, cost 14.52 s 2023-09-04 04:53:32,208 44k INFO ====> Epoch: 569, cost 14.16 s 2023-09-04 04:53:46,740 44k INFO ====> Epoch: 570, cost 14.53 s 2023-09-04 04:54:01,627 44k INFO ====> Epoch: 571, cost 14.89 s 2023-09-04 04:54:08,123 44k INFO Train Epoch: 572 [38%] 2023-09-04 04:54:08,124 44k INFO Losses: [2.3352856636047363, 2.502354860305786, 6.871198654174805, 15.887614250183105, 0.8707923293113708], step: 12000, lr: 9.311085054309703e-05, reference_loss: 28.467247009277344 2023-09-04 04:54:13,486 44k INFO Saving model and optimizer state at iteration 572 to ./logs\44k\G_12000.pth 2023-09-04 04:54:14,124 44k INFO Saving model and optimizer state at iteration 572 to ./logs\44k\D_12000.pth 2023-09-04 04:54:15,322 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_9600.pth 2023-09-04 04:54:15,355 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_9600.pth 2023-09-04 04:54:23,615 44k INFO ====> Epoch: 572, cost 21.99 s 2023-09-04 04:54:38,026 44k INFO ====> Epoch: 573, cost 14.41 s 2023-09-04 04:54:52,596 44k INFO ====> Epoch: 574, cost 14.57 s 2023-09-04 04:55:07,262 44k INFO ====> Epoch: 575, cost 14.67 s 2023-09-04 04:55:21,500 44k INFO ====> Epoch: 576, cost 14.24 s 2023-09-04 04:55:35,627 44k INFO ====> Epoch: 577, cost 14.13 s 2023-09-04 04:55:50,116 44k INFO ====> Epoch: 578, cost 14.49 s 2023-09-04 04:56:04,842 44k INFO ====> Epoch: 579, cost 14.73 s 2023-09-04 04:56:19,482 44k INFO ====> Epoch: 580, cost 14.64 s 2023-09-04 04:56:33,533 44k INFO Train Epoch: 581 [90%] 2023-09-04 04:56:33,534 44k INFO Losses: [2.2967939376831055, 2.561243772506714, 7.339554786682129, 18.019065856933594, 0.9554201364517212], step: 12200, lr: 9.300615319581631e-05, reference_loss: 31.17207908630371 2023-09-04 04:56:34,186 44k INFO ====> Epoch: 581, cost 14.70 s 2023-09-04 04:56:48,620 44k INFO ====> Epoch: 582, cost 14.43 s 2023-09-04 04:57:03,082 44k INFO ====> Epoch: 583, cost 14.46 s 2023-09-04 04:57:17,495 44k INFO ====> Epoch: 584, cost 14.41 s 2023-09-04 04:57:31,764 44k INFO ====> Epoch: 585, cost 14.27 s 2023-09-04 04:57:46,102 44k INFO ====> Epoch: 586, cost 14.34 s 2023-09-04 04:58:00,542 44k INFO ====> Epoch: 587, cost 14.44 s 2023-09-04 04:58:15,025 44k INFO ====> Epoch: 588, cost 14.48 s 2023-09-04 04:58:29,262 44k INFO ====> Epoch: 589, cost 14.24 s 2023-09-04 04:58:43,487 44k INFO ====> Epoch: 590, cost 14.23 s 2023-09-04 04:58:50,711 44k INFO Train Epoch: 591 [43%] 2023-09-04 04:58:50,712 44k INFO Losses: [2.3394665718078613, 2.5401737689971924, 5.970667362213135, 15.906972885131836, 0.7127355933189392], step: 12400, lr: 9.288996087747943e-05, reference_loss: 27.470016479492188 2023-09-04 04:58:58,506 44k INFO ====> Epoch: 591, cost 15.02 s 2023-09-04 04:59:12,913 44k INFO ====> Epoch: 592, cost 14.41 s 2023-09-04 04:59:27,177 44k INFO ====> Epoch: 593, cost 14.26 s 2023-09-04 04:59:41,473 44k INFO ====> Epoch: 594, cost 14.30 s 2023-09-04 04:59:56,317 44k INFO ====> Epoch: 595, cost 14.84 s 2023-09-04 05:00:10,818 44k INFO ====> Epoch: 596, cost 14.50 s 2023-09-04 05:00:25,313 44k INFO ====> Epoch: 597, cost 14.50 s 2023-09-04 05:00:39,755 44k INFO ====> Epoch: 598, cost 14.44 s 2023-09-04 05:00:54,499 44k INFO ====> Epoch: 599, cost 14.74 s 2023-09-04 05:01:09,254 44k INFO Train Epoch: 600 [95%] 2023-09-04 05:01:09,254 44k INFO Losses: [2.321568250656128, 2.499690294265747, 7.747135162353516, 16.355224609375, 0.5927027463912964], step: 12600, lr: 9.27855119068583e-05, reference_loss: 29.516321182250977 2023-09-04 05:01:09,514 44k INFO ====> Epoch: 600, cost 15.01 s 2023-09-04 05:01:23,838 44k INFO ====> Epoch: 601, cost 14.32 s 2023-09-04 05:01:38,065 44k INFO ====> Epoch: 602, cost 14.23 s 2023-09-04 05:01:52,618 44k INFO ====> Epoch: 603, cost 14.55 s 2023-09-04 05:02:07,211 44k INFO ====> Epoch: 604, cost 14.59 s 2023-09-04 05:02:21,841 44k INFO ====> Epoch: 605, cost 14.63 s 2023-09-04 05:02:36,568 44k INFO ====> Epoch: 606, cost 14.73 s 2023-09-04 05:02:51,284 44k INFO ====> Epoch: 607, cost 14.72 s 2023-09-04 05:03:05,748 44k INFO ====> Epoch: 608, cost 14.46 s 2023-09-04 05:03:20,101 44k INFO ====> Epoch: 609, cost 14.35 s 2023-09-04 05:03:27,642 44k INFO Train Epoch: 610 [48%] 2023-09-04 05:03:27,643 44k INFO Losses: [2.2871689796447754, 2.6004080772399902, 7.880055904388428, 17.89446258544922, 0.8246519565582275], step: 12800, lr: 9.266959523504591e-05, reference_loss: 31.48674774169922 2023-09-04 05:03:32,994 44k INFO Saving model and optimizer state at iteration 610 to ./logs\44k\G_12800.pth 2023-09-04 05:03:33,810 44k INFO Saving model and optimizer state at iteration 610 to ./logs\44k\D_12800.pth 2023-09-04 05:03:34,912 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_10400.pth 2023-09-04 05:03:34,945 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_10400.pth 2023-09-04 05:03:41,844 44k INFO ====> Epoch: 610, cost 21.74 s 2023-09-04 05:03:56,471 44k INFO ====> Epoch: 611, cost 14.63 s 2023-09-04 05:04:11,010 44k INFO ====> Epoch: 612, cost 14.54 s 2023-09-04 05:04:25,337 44k INFO ====> Epoch: 613, cost 14.33 s 2023-09-04 05:04:39,789 44k INFO ====> Epoch: 614, cost 14.45 s 2023-09-04 05:04:54,564 44k INFO ====> Epoch: 615, cost 14.78 s 2023-09-04 05:05:09,216 44k INFO ====> Epoch: 616, cost 14.65 s 2023-09-04 05:05:23,340 44k INFO ====> Epoch: 617, cost 14.12 s 2023-09-04 05:05:37,593 44k INFO ====> Epoch: 618, cost 14.25 s 2023-09-04 05:05:52,590 44k INFO ====> Epoch: 619, cost 15.00 s 2023-09-04 05:05:53,353 44k INFO Train Epoch: 620 [0%] 2023-09-04 05:05:53,353 44k INFO Losses: [2.4799718856811523, 2.2541136741638184, 6.321267604827881, 16.140304565429688, 0.7146345376968384], step: 13000, lr: 9.255382337759651e-05, reference_loss: 27.91029167175293 2023-09-04 05:06:08,080 44k INFO ====> Epoch: 620, cost 15.49 s 2023-09-04 05:06:22,487 44k INFO ====> Epoch: 621, cost 14.41 s 2023-09-04 05:06:36,988 44k INFO ====> Epoch: 622, cost 14.50 s 2023-09-04 05:06:51,730 44k INFO ====> Epoch: 623, cost 14.74 s 2023-09-04 05:07:06,343 44k INFO ====> Epoch: 624, cost 14.61 s 2023-09-04 05:07:20,661 44k INFO ====> Epoch: 625, cost 14.32 s 2023-09-04 05:07:35,009 44k INFO ====> Epoch: 626, cost 14.35 s 2023-09-04 05:07:49,663 44k INFO ====> Epoch: 627, cost 14.65 s 2023-09-04 05:08:04,282 44k INFO ====> Epoch: 628, cost 14.62 s 2023-09-04 05:08:12,836 44k INFO Train Epoch: 629 [52%] 2023-09-04 05:08:12,837 44k INFO Losses: [2.546570301055908, 2.6460208892822266, 5.652454376220703, 17.50933074951172, 0.8588024973869324], step: 13200, lr: 9.244975237264057e-05, reference_loss: 29.213178634643555 2023-09-04 05:08:19,231 44k INFO ====> Epoch: 629, cost 14.95 s 2023-09-04 05:08:33,698 44k INFO ====> Epoch: 630, cost 14.47 s 2023-09-04 05:08:48,344 44k INFO ====> Epoch: 631, cost 14.65 s 2023-09-04 05:09:02,957 44k INFO ====> Epoch: 632, cost 14.61 s 2023-09-04 05:09:17,345 44k INFO ====> Epoch: 633, cost 14.39 s 2023-09-04 05:09:31,486 44k INFO ====> Epoch: 634, cost 14.14 s 2023-09-04 05:09:46,175 44k INFO ====> Epoch: 635, cost 14.69 s 2023-09-04 05:10:00,643 44k INFO ====> Epoch: 636, cost 14.47 s 2023-09-04 05:10:15,158 44k INFO ====> Epoch: 637, cost 14.51 s 2023-09-04 05:10:29,534 44k INFO ====> Epoch: 638, cost 14.38 s 2023-09-04 05:10:30,918 44k INFO Train Epoch: 639 [5%] 2023-09-04 05:10:30,919 44k INFO Losses: [2.3521180152893066, 2.6102476119995117, 6.215083599090576, 18.405319213867188, 0.6307048201560974], step: 13400, lr: 9.233425516424368e-05, reference_loss: 30.213472366333008 2023-09-04 05:10:44,382 44k INFO ====> Epoch: 639, cost 14.85 s 2023-09-04 05:10:59,147 44k INFO ====> Epoch: 640, cost 14.77 s 2023-09-04 05:11:13,711 44k INFO ====> Epoch: 641, cost 14.56 s 2023-09-04 05:11:27,967 44k INFO ====> Epoch: 642, cost 14.26 s 2023-09-04 05:11:42,428 44k INFO ====> Epoch: 643, cost 14.46 s 2023-09-04 05:11:57,215 44k INFO ====> Epoch: 644, cost 14.79 s 2023-09-04 05:12:11,806 44k INFO ====> Epoch: 645, cost 14.59 s 2023-09-04 05:12:26,281 44k INFO ====> Epoch: 646, cost 14.48 s 2023-09-04 05:12:40,876 44k INFO ====> Epoch: 647, cost 14.59 s 2023-09-04 05:12:50,093 44k INFO Train Epoch: 648 [57%] 2023-09-04 05:12:50,093 44k INFO Losses: [2.409543752670288, 2.4673562049865723, 6.355193138122559, 16.02578353881836, 0.9340925216674805], step: 13600, lr: 9.223043105005667e-05, reference_loss: 28.191970825195312 2023-09-04 05:12:55,642 44k INFO Saving model and optimizer state at iteration 648 to ./logs\44k\G_13600.pth 2023-09-04 05:12:56,265 44k INFO Saving model and optimizer state at iteration 648 to ./logs\44k\D_13600.pth 2023-09-04 05:12:57,510 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_11200.pth 2023-09-04 05:12:57,540 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_11200.pth 2023-09-04 05:13:02,993 44k INFO ====> Epoch: 648, cost 22.12 s 2023-09-04 05:13:17,431 44k INFO ====> Epoch: 649, cost 14.44 s 2023-09-04 05:13:31,727 44k INFO ====> Epoch: 650, cost 14.30 s 2023-09-04 05:13:46,087 44k INFO ====> Epoch: 651, cost 14.36 s 2023-09-04 05:14:00,664 44k INFO ====> Epoch: 652, cost 14.58 s 2023-09-04 05:14:15,234 44k INFO ====> Epoch: 653, cost 14.57 s 2023-09-04 05:14:29,779 44k INFO ====> Epoch: 654, cost 14.54 s 2023-09-04 05:14:44,206 44k INFO ====> Epoch: 655, cost 14.43 s 2023-09-04 05:14:58,644 44k INFO ====> Epoch: 656, cost 14.44 s 2023-09-04 05:15:13,063 44k INFO ====> Epoch: 657, cost 14.42 s 2023-09-04 05:15:15,114 44k INFO Train Epoch: 658 [10%] 2023-09-04 05:15:15,114 44k INFO Losses: [2.3650546073913574, 2.4868757724761963, 6.723273277282715, 15.303080558776855, 0.8250734806060791], step: 13800, lr: 9.211520783915413e-05, reference_loss: 27.703357696533203 2023-09-04 05:15:27,726 44k INFO ====> Epoch: 658, cost 14.66 s 2023-09-04 05:15:41,957 44k INFO ====> Epoch: 659, cost 14.23 s 2023-09-04 05:15:56,387 44k INFO ====> Epoch: 660, cost 14.43 s 2023-09-04 05:16:10,664 44k INFO ====> Epoch: 661, cost 14.28 s 2023-09-04 05:16:25,277 44k INFO ====> Epoch: 662, cost 14.61 s 2023-09-04 05:16:39,554 44k INFO ====> Epoch: 663, cost 14.28 s 2023-09-04 05:16:54,234 44k INFO ====> Epoch: 664, cost 14.68 s 2023-09-04 05:17:08,738 44k INFO ====> Epoch: 665, cost 14.50 s 2023-09-04 05:17:23,163 44k INFO ====> Epoch: 666, cost 14.42 s 2023-09-04 05:17:32,812 44k INFO Train Epoch: 667 [62%] 2023-09-04 05:17:32,813 44k INFO Losses: [2.377427339553833, 2.5551438331604004, 7.614535808563232, 18.85548973083496, 0.7293151021003723], step: 14000, lr: 9.201163003002964e-05, reference_loss: 32.13191223144531 2023-09-04 05:17:37,711 44k INFO ====> Epoch: 667, cost 14.55 s 2023-09-04 05:17:52,559 44k INFO ====> Epoch: 668, cost 14.85 s 2023-09-04 05:18:07,614 44k INFO ====> Epoch: 669, cost 15.05 s 2023-09-04 05:18:22,146 44k INFO ====> Epoch: 670, cost 14.53 s 2023-09-04 05:18:36,654 44k INFO ====> Epoch: 671, cost 14.51 s 2023-09-04 05:18:51,197 44k INFO ====> Epoch: 672, cost 14.54 s 2023-09-04 05:19:05,883 44k INFO ====> Epoch: 673, cost 14.69 s 2023-09-04 05:19:20,294 44k INFO ====> Epoch: 674, cost 14.41 s 2023-09-04 05:19:34,607 44k INFO ====> Epoch: 675, cost 14.31 s 2023-09-04 05:19:49,188 44k INFO ====> Epoch: 676, cost 14.58 s 2023-09-04 05:19:52,016 44k INFO Train Epoch: 677 [14%] 2023-09-04 05:19:52,017 44k INFO Losses: [2.4490811824798584, 2.262295961380005, 6.3970184326171875, 17.05535316467285, 0.7331574559211731], step: 14200, lr: 9.189668016660891e-05, reference_loss: 28.896907806396484 2023-09-04 05:20:04,028 44k INFO ====> Epoch: 677, cost 14.84 s 2023-09-04 05:20:18,382 44k INFO ====> Epoch: 678, cost 14.35 s 2023-09-04 05:20:32,832 44k INFO ====> Epoch: 679, cost 14.45 s 2023-09-04 05:20:47,318 44k INFO ====> Epoch: 680, cost 14.49 s 2023-09-04 05:21:02,048 44k INFO ====> Epoch: 681, cost 14.73 s 2023-09-04 05:21:16,517 44k INFO ====> Epoch: 682, cost 14.47 s 2023-09-04 05:21:30,834 44k INFO ====> Epoch: 683, cost 14.32 s 2023-09-04 05:21:45,295 44k INFO ====> Epoch: 684, cost 14.46 s 2023-09-04 05:21:59,837 44k INFO ====> Epoch: 685, cost 14.54 s 2023-09-04 05:22:10,489 44k INFO Train Epoch: 686 [67%] 2023-09-04 05:22:10,489 44k INFO Losses: [2.402996301651001, 2.341766834259033, 6.441955089569092, 18.04885482788086, 0.7186977863311768], step: 14400, lr: 9.179334807823006e-05, reference_loss: 29.95427131652832 2023-09-04 05:22:15,832 44k INFO Saving model and optimizer state at iteration 686 to ./logs\44k\G_14400.pth 2023-09-04 05:22:16,381 44k INFO Saving model and optimizer state at iteration 686 to ./logs\44k\D_14400.pth 2023-09-04 05:22:17,728 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_12000.pth 2023-09-04 05:22:17,763 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_12000.pth 2023-09-04 05:22:21,667 44k INFO ====> Epoch: 686, cost 21.83 s 2023-09-04 05:22:35,928 44k INFO ====> Epoch: 687, cost 14.26 s 2023-09-04 05:22:50,526 44k INFO ====> Epoch: 688, cost 14.60 s 2023-09-04 05:23:05,004 44k INFO ====> Epoch: 689, cost 14.48 s 2023-09-04 05:23:19,323 44k INFO ====> Epoch: 690, cost 14.32 s 2023-09-04 05:23:33,553 44k INFO ====> Epoch: 691, cost 14.23 s 2023-09-04 05:23:48,128 44k INFO ====> Epoch: 692, cost 14.57 s 2023-09-04 05:24:02,896 44k INFO ====> Epoch: 693, cost 14.77 s 2023-09-04 05:24:17,646 44k INFO ====> Epoch: 694, cost 14.75 s 2023-09-04 05:24:31,920 44k INFO ====> Epoch: 695, cost 14.27 s 2023-09-04 05:24:35,410 44k INFO Train Epoch: 696 [19%] 2023-09-04 05:24:35,411 44k INFO Losses: [2.540642499923706, 2.204317092895508, 5.733966827392578, 17.07329750061035, 0.7784384489059448], step: 14600, lr: 9.167867091382074e-05, reference_loss: 28.33066177368164 2023-09-04 05:24:46,939 44k INFO ====> Epoch: 696, cost 15.02 s 2023-09-04 05:25:01,605 44k INFO ====> Epoch: 697, cost 14.67 s 2023-09-04 05:25:16,411 44k INFO ====> Epoch: 698, cost 14.81 s 2023-09-04 05:25:30,593 44k INFO ====> Epoch: 699, cost 14.18 s 2023-09-04 05:25:45,128 44k INFO ====> Epoch: 700, cost 14.54 s 2023-09-04 05:25:59,805 44k INFO ====> Epoch: 701, cost 14.68 s 2023-09-04 05:26:14,445 44k INFO ====> Epoch: 702, cost 14.64 s 2023-09-04 05:26:28,824 44k INFO ====> Epoch: 703, cost 14.38 s 2023-09-04 05:26:43,226 44k INFO ====> Epoch: 704, cost 14.40 s 2023-09-04 05:26:54,480 44k INFO Train Epoch: 705 [71%] 2023-09-04 05:26:54,481 44k INFO Losses: [2.1641101837158203, 2.849407196044922, 7.122254371643066, 15.591399192810059, 0.6997323632240295], step: 14800, lr: 9.157558396325682e-05, reference_loss: 28.426902770996094 2023-09-04 05:26:57,983 44k INFO ====> Epoch: 705, cost 14.76 s 2023-09-04 05:27:12,443 44k INFO ====> Epoch: 706, cost 14.46 s 2023-09-04 05:27:26,665 44k INFO ====> Epoch: 707, cost 14.22 s 2023-09-04 05:27:41,114 44k INFO ====> Epoch: 708, cost 14.45 s 2023-09-04 05:27:55,512 44k INFO ====> Epoch: 709, cost 14.40 s 2023-09-04 05:28:10,242 44k INFO ====> Epoch: 710, cost 14.73 s 2023-09-04 05:28:24,530 44k INFO ====> Epoch: 711, cost 14.29 s 2023-09-04 05:28:38,914 44k INFO ====> Epoch: 712, cost 14.38 s 2023-09-04 05:28:53,630 44k INFO ====> Epoch: 713, cost 14.72 s 2023-09-04 05:29:08,298 44k INFO ====> Epoch: 714, cost 14.67 s 2023-09-04 05:29:12,521 44k INFO Train Epoch: 715 [24%] 2023-09-04 05:29:12,522 44k INFO Losses: [2.3971242904663086, 2.5196590423583984, 6.938624858856201, 15.800455093383789, 0.7026923894882202], step: 15000, lr: 9.146117885092685e-05, reference_loss: 28.35855484008789 2023-09-04 05:29:22,764 44k INFO ====> Epoch: 715, cost 14.47 s 2023-09-04 05:29:36,948 44k INFO ====> Epoch: 716, cost 14.18 s 2023-09-04 05:29:51,901 44k INFO ====> Epoch: 717, cost 14.95 s 2023-09-04 05:30:06,784 44k INFO ====> Epoch: 718, cost 14.88 s 2023-09-04 05:30:21,310 44k INFO ====> Epoch: 719, cost 14.53 s 2023-09-04 05:30:35,888 44k INFO ====> Epoch: 720, cost 14.58 s 2023-09-04 05:30:50,469 44k INFO ====> Epoch: 721, cost 14.58 s 2023-09-04 05:31:05,291 44k INFO ====> Epoch: 722, cost 14.82 s 2023-09-04 05:31:19,892 44k INFO ====> Epoch: 723, cost 14.60 s 2023-09-04 05:31:31,532 44k INFO Train Epoch: 724 [76%] 2023-09-04 05:31:31,532 44k INFO Losses: [2.353830575942993, 2.4450411796569824, 8.213234901428223, 19.21962547302246, 0.8088536858558655], step: 15200, lr: 9.13583364566301e-05, reference_loss: 33.040584564208984 2023-09-04 05:31:36,958 44k INFO Saving model and optimizer state at iteration 724 to ./logs\44k\G_15200.pth 2023-09-04 05:31:37,548 44k INFO Saving model and optimizer state at iteration 724 to ./logs\44k\D_15200.pth 2023-09-04 05:31:38,773 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_12800.pth 2023-09-04 05:31:38,806 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_12800.pth 2023-09-04 05:31:41,422 44k INFO ====> Epoch: 724, cost 21.53 s 2023-09-04 05:31:56,413 44k INFO ====> Epoch: 725, cost 14.99 s 2023-09-04 05:32:10,898 44k INFO ====> Epoch: 726, cost 14.49 s 2023-09-04 05:32:25,210 44k INFO ====> Epoch: 727, cost 14.31 s 2023-09-04 05:32:39,529 44k INFO ====> Epoch: 728, cost 14.32 s 2023-09-04 05:32:54,284 44k INFO ====> Epoch: 729, cost 14.75 s 2023-09-04 05:33:08,730 44k INFO ====> Epoch: 730, cost 14.45 s 2023-09-04 05:33:23,055 44k INFO ====> Epoch: 731, cost 14.33 s 2023-09-04 05:33:37,233 44k INFO ====> Epoch: 732, cost 14.18 s 2023-09-04 05:33:51,520 44k INFO ====> Epoch: 733, cost 14.29 s 2023-09-04 05:33:56,505 44k INFO Train Epoch: 734 [29%] 2023-09-04 05:33:56,505 44k INFO Losses: [2.5026519298553467, 2.316138982772827, 5.28476619720459, 15.50872802734375, 0.8045852184295654], step: 15400, lr: 9.124420275098216e-05, reference_loss: 26.4168701171875 2023-09-04 05:34:06,383 44k INFO ====> Epoch: 734, cost 14.86 s 2023-09-04 05:34:20,833 44k INFO ====> Epoch: 735, cost 14.45 s 2023-09-04 05:34:35,202 44k INFO ====> Epoch: 736, cost 14.37 s 2023-09-04 05:34:49,858 44k INFO ====> Epoch: 737, cost 14.66 s 2023-09-04 05:35:04,455 44k INFO ====> Epoch: 738, cost 14.60 s 2023-09-04 05:35:18,837 44k INFO ====> Epoch: 739, cost 14.38 s 2023-09-04 05:35:33,073 44k INFO ====> Epoch: 740, cost 14.24 s 2023-09-04 05:35:47,557 44k INFO ====> Epoch: 741, cost 14.48 s 2023-09-04 05:36:02,393 44k INFO ====> Epoch: 742, cost 14.84 s 2023-09-04 05:36:15,115 44k INFO Train Epoch: 743 [81%] 2023-09-04 05:36:15,116 44k INFO Losses: [2.353813409805298, 2.369408130645752, 7.255029678344727, 16.489227294921875, 0.7263268232345581], step: 15600, lr: 9.114160433278438e-05, reference_loss: 29.193805694580078 2023-09-04 05:36:17,206 44k INFO ====> Epoch: 743, cost 14.81 s 2023-09-04 05:36:31,727 44k INFO ====> Epoch: 744, cost 14.52 s 2023-09-04 05:36:46,413 44k INFO ====> Epoch: 745, cost 14.69 s 2023-09-04 05:37:01,094 44k INFO ====> Epoch: 746, cost 14.68 s 2023-09-04 05:37:15,516 44k INFO ====> Epoch: 747, cost 14.42 s 2023-09-04 05:37:29,761 44k INFO ====> Epoch: 748, cost 14.24 s 2023-09-04 05:37:44,184 44k INFO ====> Epoch: 749, cost 14.42 s 2023-09-04 05:37:58,936 44k INFO ====> Epoch: 750, cost 14.75 s 2023-09-04 05:38:13,538 44k INFO ====> Epoch: 751, cost 14.60 s 2023-09-04 05:38:28,036 44k INFO ====> Epoch: 752, cost 14.50 s 2023-09-04 05:38:33,643 44k INFO Train Epoch: 753 [33%] 2023-09-04 05:38:33,643 44k INFO Losses: [2.4154868125915527, 2.362367868423462, 5.437875270843506, 16.48991584777832, 0.8300819993019104], step: 15800, lr: 9.102774138995226e-05, reference_loss: 27.535728454589844 2023-09-04 05:38:42,659 44k INFO ====> Epoch: 753, cost 14.62 s 2023-09-04 05:38:57,215 44k INFO ====> Epoch: 754, cost 14.56 s 2023-09-04 05:39:11,754 44k INFO ====> Epoch: 755, cost 14.54 s 2023-09-04 05:39:26,073 44k INFO ====> Epoch: 756, cost 14.32 s 2023-09-04 05:39:40,255 44k INFO ====> Epoch: 757, cost 14.18 s 2023-09-04 05:39:54,776 44k INFO ====> Epoch: 758, cost 14.52 s 2023-09-04 05:40:09,294 44k INFO ====> Epoch: 759, cost 14.52 s 2023-09-04 05:40:23,637 44k INFO ====> Epoch: 760, cost 14.34 s 2023-09-04 05:40:37,972 44k INFO ====> Epoch: 761, cost 14.34 s 2023-09-04 05:40:51,503 44k INFO Train Epoch: 762 [86%] 2023-09-04 05:40:51,503 44k INFO Losses: [2.40901517868042, 2.50052809715271, 8.314818382263184, 17.84373664855957, 0.7530693411827087], step: 16000, lr: 9.092538636906162e-05, reference_loss: 31.8211669921875 2023-09-04 05:40:56,947 44k INFO Saving model and optimizer state at iteration 762 to ./logs\44k\G_16000.pth 2023-09-04 05:40:58,248 44k INFO Saving model and optimizer state at iteration 762 to ./logs\44k\D_16000.pth 2023-09-04 05:40:59,323 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_13600.pth 2023-09-04 05:40:59,354 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_13600.pth 2023-09-04 05:41:00,469 44k INFO ====> Epoch: 762, cost 22.50 s 2023-09-04 05:41:14,912 44k INFO ====> Epoch: 763, cost 14.44 s 2023-09-04 05:41:29,127 44k INFO ====> Epoch: 764, cost 14.21 s 2023-09-04 05:41:43,812 44k INFO ====> Epoch: 765, cost 14.69 s 2023-09-04 05:41:58,710 44k INFO ====> Epoch: 766, cost 14.90 s 2023-09-04 05:42:13,406 44k INFO ====> Epoch: 767, cost 14.70 s 2023-09-04 05:42:27,852 44k INFO ====> Epoch: 768, cost 14.45 s 2023-09-04 05:42:42,393 44k INFO ====> Epoch: 769, cost 14.54 s 2023-09-04 05:42:57,077 44k INFO ====> Epoch: 770, cost 14.68 s 2023-09-04 05:43:11,910 44k INFO ====> Epoch: 771, cost 14.83 s 2023-09-04 05:43:18,147 44k INFO Train Epoch: 772 [38%] 2023-09-04 05:43:18,147 44k INFO Losses: [2.3478362560272217, 2.407737970352173, 7.815682411193848, 17.758495330810547, 0.7258373498916626], step: 16200, lr: 9.081179354670654e-05, reference_loss: 31.05558967590332 2023-09-04 05:43:26,383 44k INFO ====> Epoch: 772, cost 14.47 s 2023-09-04 05:43:40,581 44k INFO ====> Epoch: 773, cost 14.20 s 2023-09-04 05:43:55,166 44k INFO ====> Epoch: 774, cost 14.58 s 2023-09-04 05:44:09,878 44k INFO ====> Epoch: 775, cost 14.71 s 2023-09-04 05:44:24,404 44k INFO ====> Epoch: 776, cost 14.53 s 2023-09-04 05:44:38,872 44k INFO ====> Epoch: 777, cost 14.47 s 2023-09-04 05:44:53,497 44k INFO ====> Epoch: 778, cost 14.63 s 2023-09-04 05:45:07,885 44k INFO ====> Epoch: 779, cost 14.39 s 2023-09-04 05:45:22,291 44k INFO ====> Epoch: 780, cost 14.41 s 2023-09-04 05:45:36,159 44k INFO Train Epoch: 781 [90%] 2023-09-04 05:45:36,159 44k INFO Losses: [2.408935308456421, 2.3101119995117188, 6.430037975311279, 16.930761337280273, 0.6319032311439514], step: 16400, lr: 9.07096813457043e-05, reference_loss: 28.711750030517578 2023-09-04 05:45:36,834 44k INFO ====> Epoch: 781, cost 14.54 s 2023-09-04 05:45:51,296 44k INFO ====> Epoch: 782, cost 14.46 s 2023-09-04 05:46:05,985 44k INFO ====> Epoch: 783, cost 14.69 s 2023-09-04 05:46:20,044 44k INFO ====> Epoch: 784, cost 14.06 s 2023-09-04 05:46:34,381 44k INFO ====> Epoch: 785, cost 14.34 s 2023-09-04 05:46:48,902 44k INFO ====> Epoch: 786, cost 14.52 s 2023-09-04 05:47:03,597 44k INFO ====> Epoch: 787, cost 14.70 s 2023-09-04 05:47:17,990 44k INFO ====> Epoch: 788, cost 14.39 s 2023-09-04 05:47:32,242 44k INFO ====> Epoch: 789, cost 14.25 s 2023-09-04 05:47:46,726 44k INFO ====> Epoch: 790, cost 14.48 s 2023-09-04 05:47:53,903 44k INFO Train Epoch: 791 [43%] 2023-09-04 05:47:53,903 44k INFO Losses: [2.311046838760376, 2.407803535461426, 7.671084403991699, 17.56828498840332, 0.7485687732696533], step: 16600, lr: 9.059635800301143e-05, reference_loss: 30.706789016723633 2023-09-04 05:48:01,782 44k INFO ====> Epoch: 791, cost 15.06 s 2023-09-04 05:48:16,288 44k INFO ====> Epoch: 792, cost 14.51 s 2023-09-04 05:48:30,779 44k INFO ====> Epoch: 793, cost 14.49 s 2023-09-04 05:48:45,475 44k INFO ====> Epoch: 794, cost 14.70 s 2023-09-04 05:49:00,277 44k INFO ====> Epoch: 795, cost 14.80 s 2023-09-04 05:49:14,821 44k INFO ====> Epoch: 796, cost 14.54 s 2023-09-04 05:49:29,193 44k INFO ====> Epoch: 797, cost 14.37 s 2023-09-04 05:49:43,521 44k INFO ====> Epoch: 798, cost 14.33 s 2023-09-04 05:49:58,173 44k INFO ====> Epoch: 799, cost 14.65 s 2023-09-04 05:50:12,812 44k INFO Train Epoch: 800 [95%] 2023-09-04 05:50:12,812 44k INFO Losses: [2.26790452003479, 2.6383907794952393, 6.796956539154053, 17.960742950439453, 0.4294731616973877], step: 16800, lr: 9.049448804584871e-05, reference_loss: 30.093467712402344 2023-09-04 05:50:18,156 44k INFO Saving model and optimizer state at iteration 800 to ./logs\44k\G_16800.pth 2023-09-04 05:50:18,778 44k INFO Saving model and optimizer state at iteration 800 to ./logs\44k\D_16800.pth 2023-09-04 05:50:19,957 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_14400.pth 2023-09-04 05:50:19,990 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_14400.pth 2023-09-04 05:50:19,990 44k INFO ====> Epoch: 800, cost 21.82 s 2023-09-04 05:50:34,439 44k INFO ====> Epoch: 801, cost 14.45 s 2023-09-04 05:50:48,884 44k INFO ====> Epoch: 802, cost 14.44 s 2023-09-04 05:51:03,498 44k INFO ====> Epoch: 803, cost 14.61 s 2023-09-04 05:51:18,039 44k INFO ====> Epoch: 804, cost 14.54 s 2023-09-04 05:51:32,410 44k INFO ====> Epoch: 805, cost 14.37 s 2023-09-04 05:51:46,965 44k INFO ====> Epoch: 806, cost 14.56 s 2023-09-04 05:52:01,551 44k INFO ====> Epoch: 807, cost 14.59 s 2023-09-04 05:52:16,111 44k INFO ====> Epoch: 808, cost 14.56 s 2023-09-04 05:52:30,494 44k INFO ====> Epoch: 809, cost 14.38 s 2023-09-04 05:52:38,212 44k INFO Train Epoch: 810 [48%] 2023-09-04 05:52:38,212 44k INFO Losses: [2.117483615875244, 2.820739269256592, 8.191574096679688, 17.807199478149414, 0.8307142853736877], step: 17000, lr: 9.038143354352324e-05, reference_loss: 31.767711639404297 2023-09-04 05:52:45,277 44k INFO ====> Epoch: 810, cost 14.78 s 2023-09-04 05:52:59,884 44k INFO ====> Epoch: 811, cost 14.61 s 2023-09-04 05:53:14,372 44k INFO ====> Epoch: 812, cost 14.49 s 2023-09-04 05:53:28,633 44k INFO ====> Epoch: 813, cost 14.26 s 2023-09-04 05:53:43,083 44k INFO ====> Epoch: 814, cost 14.45 s 2023-09-04 05:53:57,973 44k INFO ====> Epoch: 815, cost 14.89 s 2023-09-04 05:54:12,620 44k INFO ====> Epoch: 816, cost 14.65 s 2023-09-04 05:54:27,164 44k INFO ====> Epoch: 817, cost 14.54 s 2023-09-04 05:54:41,736 44k INFO ====> Epoch: 818, cost 14.57 s 2023-09-04 05:54:56,558 44k INFO ====> Epoch: 819, cost 14.82 s 2023-09-04 05:54:57,250 44k INFO Train Epoch: 820 [0%] 2023-09-04 05:54:57,251 44k INFO Losses: [2.2777223587036133, 2.4313197135925293, 6.84441614151001, 15.98019790649414, 0.7506565451622009], step: 17200, lr: 9.026852027986074e-05, reference_loss: 28.284313201904297 2023-09-04 05:55:11,353 44k INFO ====> Epoch: 820, cost 14.80 s 2023-09-04 05:55:25,486 44k INFO ====> Epoch: 821, cost 14.13 s 2023-09-04 05:55:39,602 44k INFO ====> Epoch: 822, cost 14.12 s 2023-09-04 05:55:54,273 44k INFO ====> Epoch: 823, cost 14.67 s 2023-09-04 05:56:09,150 44k INFO ====> Epoch: 824, cost 14.88 s 2023-09-04 05:56:23,612 44k INFO ====> Epoch: 825, cost 14.46 s 2023-09-04 05:56:38,046 44k INFO ====> Epoch: 826, cost 14.43 s 2023-09-04 05:56:52,610 44k INFO ====> Epoch: 827, cost 14.56 s 2023-09-04 05:57:07,105 44k INFO ====> Epoch: 828, cost 14.49 s 2023-09-04 05:57:15,415 44k INFO Train Epoch: 829 [52%] 2023-09-04 05:57:15,415 44k INFO Losses: [2.5155680179595947, 2.3709452152252197, 6.015244960784912, 16.90566635131836, 0.6186666488647461], step: 17400, lr: 9.01670189557816e-05, reference_loss: 28.426090240478516 2023-09-04 05:57:21,672 44k INFO ====> Epoch: 829, cost 14.57 s 2023-09-04 05:57:35,942 44k INFO ====> Epoch: 830, cost 14.27 s 2023-09-04 05:57:50,451 44k INFO ====> Epoch: 831, cost 14.51 s 2023-09-04 05:58:04,941 44k INFO ====> Epoch: 832, cost 14.49 s 2023-09-04 05:58:19,404 44k INFO ====> Epoch: 833, cost 14.46 s 2023-09-04 05:58:33,732 44k INFO ====> Epoch: 834, cost 14.33 s 2023-09-04 05:58:48,325 44k INFO ====> Epoch: 835, cost 14.59 s 2023-09-04 05:59:02,926 44k INFO ====> Epoch: 836, cost 14.60 s 2023-09-04 05:59:17,373 44k INFO ====> Epoch: 837, cost 14.45 s 2023-09-04 05:59:31,548 44k INFO ====> Epoch: 838, cost 14.17 s 2023-09-04 05:59:32,936 44k INFO Train Epoch: 839 [5%] 2023-09-04 05:59:32,936 44k INFO Losses: [2.5062618255615234, 2.2993197441101074, 6.231419563293457, 16.75400733947754, 0.7502939105033875], step: 17600, lr: 9.005437355964375e-05, reference_loss: 28.541301727294922 2023-09-04 05:59:38,260 44k INFO Saving model and optimizer state at iteration 839 to ./logs\44k\G_17600.pth 2023-09-04 05:59:38,851 44k INFO Saving model and optimizer state at iteration 839 to ./logs\44k\D_17600.pth 2023-09-04 05:59:40,082 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_15200.pth 2023-09-04 05:59:40,113 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_15200.pth 2023-09-04 05:59:53,584 44k INFO ====> Epoch: 839, cost 22.04 s 2023-09-04 06:00:08,461 44k INFO ====> Epoch: 840, cost 14.88 s 2023-09-04 06:00:22,836 44k INFO ====> Epoch: 841, cost 14.37 s 2023-09-04 06:00:37,211 44k INFO ====> Epoch: 842, cost 14.38 s 2023-09-04 06:00:51,921 44k INFO ====> Epoch: 843, cost 14.71 s 2023-09-04 06:01:06,857 44k INFO ====> Epoch: 844, cost 14.94 s 2023-09-04 06:01:21,416 44k INFO ====> Epoch: 845, cost 14.56 s 2023-09-04 06:01:35,732 44k INFO ====> Epoch: 846, cost 14.32 s 2023-09-04 06:01:50,432 44k INFO ====> Epoch: 847, cost 14.70 s 2023-09-04 06:01:59,682 44k INFO Train Epoch: 848 [57%] 2023-09-04 06:01:59,682 44k INFO Losses: [2.2597429752349854, 2.587305784225464, 6.617968559265137, 15.98031997680664, 0.6963230967521667], step: 17800, lr: 8.995311303020248e-05, reference_loss: 28.141660690307617 2023-09-04 06:02:05,325 44k INFO ====> Epoch: 848, cost 14.89 s 2023-09-04 06:02:19,786 44k INFO ====> Epoch: 849, cost 14.46 s 2023-09-04 06:02:34,500 44k INFO ====> Epoch: 850, cost 14.71 s 2023-09-04 06:02:49,133 44k INFO ====> Epoch: 851, cost 14.63 s 2023-09-04 06:03:03,590 44k INFO ====> Epoch: 852, cost 14.46 s 2023-09-04 06:03:18,164 44k INFO ====> Epoch: 853, cost 14.57 s 2023-09-04 06:03:32,511 44k INFO ====> Epoch: 854, cost 14.35 s 2023-09-04 06:03:47,085 44k INFO ====> Epoch: 855, cost 14.57 s 2023-09-04 06:04:01,580 44k INFO ====> Epoch: 856, cost 14.50 s 2023-09-04 06:04:15,986 44k INFO ====> Epoch: 857, cost 14.41 s 2023-09-04 06:04:18,045 44k INFO Train Epoch: 858 [10%] 2023-09-04 06:04:18,045 44k INFO Losses: [2.3777520656585693, 2.5654995441436768, 6.870899677276611, 17.119661331176758, 0.6852025985717773], step: 18000, lr: 8.984073486611914e-05, reference_loss: 29.619014739990234 2023-09-04 06:04:30,677 44k INFO ====> Epoch: 858, cost 14.69 s 2023-09-04 06:04:45,117 44k INFO ====> Epoch: 859, cost 14.44 s 2023-09-04 06:04:59,697 44k INFO ====> Epoch: 860, cost 14.58 s 2023-09-04 06:05:14,180 44k INFO ====> Epoch: 861, cost 14.48 s 2023-09-04 06:05:28,520 44k INFO ====> Epoch: 862, cost 14.34 s 2023-09-04 06:05:43,011 44k INFO ====> Epoch: 863, cost 14.49 s 2023-09-04 06:05:57,913 44k INFO ====> Epoch: 864, cost 14.90 s 2023-09-04 06:06:12,728 44k INFO ====> Epoch: 865, cost 14.82 s 2023-09-04 06:06:27,276 44k INFO ====> Epoch: 866, cost 14.55 s 2023-09-04 06:06:37,084 44k INFO Train Epoch: 867 [62%] 2023-09-04 06:06:37,084 44k INFO Losses: [2.5400071144104004, 2.2928919792175293, 7.6859049797058105, 16.734392166137695, 0.7361353635787964], step: 18200, lr: 8.973971456007135e-05, reference_loss: 29.98933219909668 2023-09-04 06:06:41,992 44k INFO ====> Epoch: 867, cost 14.72 s 2023-09-04 06:06:56,759 44k INFO ====> Epoch: 868, cost 14.77 s 2023-09-04 06:07:11,493 44k INFO ====> Epoch: 869, cost 14.73 s 2023-09-04 06:07:25,710 44k INFO ====> Epoch: 870, cost 14.22 s 2023-09-04 06:07:40,122 44k INFO ====> Epoch: 871, cost 14.41 s 2023-09-04 06:07:54,825 44k INFO ====> Epoch: 872, cost 14.70 s 2023-09-04 06:08:09,647 44k INFO ====> Epoch: 873, cost 14.82 s 2023-09-04 06:08:24,036 44k INFO ====> Epoch: 874, cost 14.39 s 2023-09-04 06:08:38,444 44k INFO ====> Epoch: 875, cost 14.41 s 2023-09-04 06:08:53,011 44k INFO ====> Epoch: 876, cost 14.57 s 2023-09-04 06:08:55,867 44k INFO Train Epoch: 877 [14%] 2023-09-04 06:08:55,868 44k INFO Losses: [2.3767895698547363, 2.53025221824646, 7.138409614562988, 17.278419494628906, 0.8102803826332092], step: 18400, lr: 8.962760299407988e-05, reference_loss: 30.134151458740234 2023-09-04 06:09:01,217 44k INFO Saving model and optimizer state at iteration 877 to ./logs\44k\G_18400.pth 2023-09-04 06:09:01,996 44k INFO Saving model and optimizer state at iteration 877 to ./logs\44k\D_18400.pth 2023-09-04 06:09:03,075 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_16000.pth 2023-09-04 06:09:03,108 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_16000.pth 2023-09-04 06:09:14,760 44k INFO ====> Epoch: 877, cost 21.75 s 2023-09-04 06:09:29,223 44k INFO ====> Epoch: 878, cost 14.46 s 2023-09-04 06:09:43,497 44k INFO ====> Epoch: 879, cost 14.27 s 2023-09-04 06:09:57,973 44k INFO ====> Epoch: 880, cost 14.48 s 2023-09-04 06:10:12,313 44k INFO ====> Epoch: 881, cost 14.34 s 2023-09-04 06:10:26,670 44k INFO ====> Epoch: 882, cost 14.36 s 2023-09-04 06:10:40,764 44k INFO ====> Epoch: 883, cost 14.09 s 2023-09-04 06:10:55,240 44k INFO ====> Epoch: 884, cost 14.48 s 2023-09-04 06:11:09,953 44k INFO ====> Epoch: 885, cost 14.71 s 2023-09-04 06:11:20,391 44k INFO Train Epoch: 886 [67%] 2023-09-04 06:11:20,392 44k INFO Losses: [2.535888910293579, 2.0105907917022705, 6.245883464813232, 15.803011894226074, 0.6745226979255676], step: 18600, lr: 8.952682234153643e-05, reference_loss: 27.2698974609375 2023-09-04 06:11:24,493 44k INFO ====> Epoch: 886, cost 14.54 s 2023-09-04 06:11:38,853 44k INFO ====> Epoch: 887, cost 14.36 s 2023-09-04 06:11:53,482 44k INFO ====> Epoch: 888, cost 14.63 s 2023-09-04 06:12:08,325 44k INFO ====> Epoch: 889, cost 14.84 s 2023-09-04 06:12:22,883 44k INFO ====> Epoch: 890, cost 14.56 s 2023-09-04 06:12:37,461 44k INFO ====> Epoch: 891, cost 14.58 s 2023-09-04 06:12:52,027 44k INFO ====> Epoch: 892, cost 14.57 s 2023-09-04 06:13:06,418 44k INFO ====> Epoch: 893, cost 14.39 s 2023-09-04 06:13:20,992 44k INFO ====> Epoch: 894, cost 14.57 s 2023-09-04 06:13:35,088 44k INFO ====> Epoch: 895, cost 14.10 s 2023-09-04 06:13:38,497 44k INFO Train Epoch: 896 [19%] 2023-09-04 06:13:38,498 44k INFO Losses: [2.4578604698181152, 2.4970693588256836, 7.297029495239258, 16.78018569946289, 0.6533662676811218], step: 18800, lr: 8.941497674117817e-05, reference_loss: 29.685510635375977 2023-09-04 06:13:49,950 44k INFO ====> Epoch: 896, cost 14.86 s 2023-09-04 06:14:04,595 44k INFO ====> Epoch: 897, cost 14.65 s 2023-09-04 06:14:18,891 44k INFO ====> Epoch: 898, cost 14.30 s 2023-09-04 06:14:33,154 44k INFO ====> Epoch: 899, cost 14.26 s 2023-09-04 06:14:47,459 44k INFO ====> Epoch: 900, cost 14.30 s 2023-09-04 06:15:02,150 44k INFO ====> Epoch: 901, cost 14.69 s 2023-09-04 06:15:16,540 44k INFO ====> Epoch: 902, cost 14.39 s 2023-09-04 06:15:30,915 44k INFO ====> Epoch: 903, cost 14.38 s 2023-09-04 06:15:45,380 44k INFO ====> Epoch: 904, cost 14.46 s 2023-09-04 06:15:56,776 44k INFO Train Epoch: 905 [71%] 2023-09-04 06:15:56,777 44k INFO Losses: [2.261328935623169, 2.6363883018493652, 6.420187950134277, 15.625527381896973, 0.5617105960845947], step: 19000, lr: 8.931443517360183e-05, reference_loss: 27.505142211914062 2023-09-04 06:16:00,192 44k INFO ====> Epoch: 905, cost 14.81 s 2023-09-04 06:16:14,652 44k INFO ====> Epoch: 906, cost 14.46 s 2023-09-04 06:16:28,999 44k INFO ====> Epoch: 907, cost 14.35 s 2023-09-04 06:16:43,523 44k INFO ====> Epoch: 908, cost 14.52 s 2023-09-04 06:16:58,183 44k INFO ====> Epoch: 909, cost 14.66 s 2023-09-04 06:17:12,920 44k INFO ====> Epoch: 910, cost 14.74 s 2023-09-04 06:17:27,193 44k INFO ====> Epoch: 911, cost 14.27 s 2023-09-04 06:17:41,586 44k INFO ====> Epoch: 912, cost 14.39 s 2023-09-04 06:17:56,556 44k INFO ====> Epoch: 913, cost 14.97 s 2023-09-04 06:18:11,506 44k INFO ====> Epoch: 914, cost 14.95 s 2023-09-04 06:18:15,684 44k INFO Train Epoch: 915 [24%] 2023-09-04 06:18:15,684 44k INFO Losses: [2.471282958984375, 2.279092788696289, 5.985932350158691, 15.165728569030762, 0.8226800560951233], step: 19200, lr: 8.920285490791852e-05, reference_loss: 26.724716186523438 2023-09-04 06:18:21,071 44k INFO Saving model and optimizer state at iteration 915 to ./logs\44k\G_19200.pth 2023-09-04 06:18:21,780 44k INFO Saving model and optimizer state at iteration 915 to ./logs\44k\D_19200.pth 2023-09-04 06:18:23,024 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_16800.pth 2023-09-04 06:18:23,056 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_16800.pth 2023-09-04 06:18:33,229 44k INFO ====> Epoch: 915, cost 21.72 s 2023-09-04 06:18:47,867 44k INFO ====> Epoch: 916, cost 14.64 s 2023-09-04 06:19:02,811 44k INFO ====> Epoch: 917, cost 14.94 s 2023-09-04 06:19:17,364 44k INFO ====> Epoch: 918, cost 14.55 s 2023-09-04 06:19:31,671 44k INFO ====> Epoch: 919, cost 14.31 s 2023-09-04 06:19:46,187 44k INFO ====> Epoch: 920, cost 14.52 s 2023-09-04 06:20:00,950 44k INFO ====> Epoch: 921, cost 14.76 s 2023-09-04 06:20:15,325 44k INFO ====> Epoch: 922, cost 14.38 s 2023-09-04 06:20:29,790 44k INFO ====> Epoch: 923, cost 14.46 s 2023-09-04 06:20:41,807 44k INFO Train Epoch: 924 [76%] 2023-09-04 06:20:41,807 44k INFO Losses: [2.405150890350342, 2.5263776779174805, 6.528800010681152, 17.44586944580078, 0.8313546180725098], step: 19400, lr: 8.910255185812085e-05, reference_loss: 29.737552642822266 2023-09-04 06:20:44,632 44k INFO ====> Epoch: 924, cost 14.84 s 2023-09-04 06:20:59,083 44k INFO ====> Epoch: 925, cost 14.45 s 2023-09-04 06:21:13,607 44k INFO ====> Epoch: 926, cost 14.52 s 2023-09-04 06:21:27,926 44k INFO ====> Epoch: 927, cost 14.32 s 2023-09-04 06:21:42,136 44k INFO ====> Epoch: 928, cost 14.21 s 2023-09-04 06:21:56,699 44k INFO ====> Epoch: 929, cost 14.56 s 2023-09-04 06:22:11,265 44k INFO ====> Epoch: 930, cost 14.57 s 2023-09-04 06:22:25,636 44k INFO ====> Epoch: 931, cost 14.37 s 2023-09-04 06:22:40,039 44k INFO ====> Epoch: 932, cost 14.40 s 2023-09-04 06:22:54,477 44k INFO ====> Epoch: 933, cost 14.44 s 2023-09-04 06:22:59,530 44k INFO Train Epoch: 934 [29%] 2023-09-04 06:22:59,530 44k INFO Losses: [2.33569598197937, 2.6318020820617676, 6.670102596282959, 16.521522521972656, 0.640150249004364], step: 19600, lr: 8.899123629765109e-05, reference_loss: 28.799272537231445 2023-09-04 06:23:09,259 44k INFO ====> Epoch: 934, cost 14.78 s 2023-09-04 06:23:23,395 44k INFO ====> Epoch: 935, cost 14.14 s 2023-09-04 06:23:37,666 44k INFO ====> Epoch: 936, cost 14.27 s 2023-09-04 06:23:52,525 44k INFO ====> Epoch: 937, cost 14.86 s 2023-09-04 06:24:07,342 44k INFO ====> Epoch: 938, cost 14.82 s 2023-09-04 06:24:21,917 44k INFO ====> Epoch: 939, cost 14.58 s 2023-09-04 06:24:36,547 44k INFO ====> Epoch: 940, cost 14.63 s 2023-09-04 06:24:51,299 44k INFO ====> Epoch: 941, cost 14.75 s 2023-09-04 06:25:06,226 44k INFO ====> Epoch: 942, cost 14.93 s 2023-09-04 06:25:18,814 44k INFO Train Epoch: 943 [81%] 2023-09-04 06:25:18,815 44k INFO Losses: [2.514463424682617, 2.2476370334625244, 5.944185733795166, 16.325939178466797, 0.8081812858581543], step: 19800, lr: 8.889117119978924e-05, reference_loss: 27.84040641784668 2023-09-04 06:25:20,887 44k INFO ====> Epoch: 943, cost 14.66 s 2023-09-04 06:25:35,105 44k INFO ====> Epoch: 944, cost 14.22 s 2023-09-04 06:25:49,567 44k INFO ====> Epoch: 945, cost 14.46 s 2023-09-04 06:26:04,401 44k INFO ====> Epoch: 946, cost 14.83 s 2023-09-04 06:26:18,935 44k INFO ====> Epoch: 947, cost 14.53 s 2023-09-04 06:26:33,393 44k INFO ====> Epoch: 948, cost 14.46 s 2023-09-04 06:26:47,891 44k INFO ====> Epoch: 949, cost 14.50 s 2023-09-04 06:27:02,513 44k INFO ====> Epoch: 950, cost 14.62 s 2023-09-04 06:27:16,992 44k INFO ====> Epoch: 951, cost 14.48 s 2023-09-04 06:27:31,296 44k INFO ====> Epoch: 952, cost 14.30 s 2023-09-04 06:27:36,822 44k INFO Train Epoch: 953 [33%] 2023-09-04 06:27:36,823 44k INFO Losses: [2.647345542907715, 2.033071279525757, 5.294334888458252, 15.855249404907227, 0.7133387923240662], step: 20000, lr: 8.87801197165649e-05, reference_loss: 26.5433406829834 2023-09-04 06:27:42,175 44k INFO Saving model and optimizer state at iteration 953 to ./logs\44k\G_20000.pth 2023-09-04 06:27:42,748 44k INFO Saving model and optimizer state at iteration 953 to ./logs\44k\D_20000.pth 2023-09-04 06:27:44,057 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_17600.pth 2023-09-04 06:27:44,089 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_17600.pth 2023-09-04 06:27:53,022 44k INFO ====> Epoch: 953, cost 21.73 s 2023-09-04 06:28:07,678 44k INFO ====> Epoch: 954, cost 14.66 s 2023-09-04 06:28:22,097 44k INFO ====> Epoch: 955, cost 14.42 s 2023-09-04 06:28:36,357 44k INFO ====> Epoch: 956, cost 14.26 s 2023-09-04 06:28:50,883 44k INFO ====> Epoch: 957, cost 14.53 s 2023-09-04 06:29:05,474 44k INFO ====> Epoch: 958, cost 14.59 s 2023-09-04 06:29:19,981 44k INFO ====> Epoch: 959, cost 14.51 s 2023-09-04 06:29:34,347 44k INFO ====> Epoch: 960, cost 14.37 s 2023-09-04 06:29:49,060 44k INFO ====> Epoch: 961, cost 14.71 s 2023-09-04 06:30:03,046 44k INFO Train Epoch: 962 [86%] 2023-09-04 06:30:03,047 44k INFO Losses: [2.3522064685821533, 2.5852231979370117, 7.111903190612793, 16.279640197753906, 0.8171075582504272], step: 20200, lr: 8.868029200613832e-05, reference_loss: 29.146080017089844 2023-09-04 06:30:04,473 44k INFO ====> Epoch: 962, cost 15.41 s 2023-09-04 06:30:19,045 44k INFO ====> Epoch: 963, cost 14.57 s 2023-09-04 06:30:33,498 44k INFO ====> Epoch: 964, cost 14.45 s 2023-09-04 06:30:48,235 44k INFO ====> Epoch: 965, cost 14.74 s 2023-09-04 06:31:03,047 44k INFO ====> Epoch: 966, cost 14.81 s 2023-09-04 06:31:17,678 44k INFO ====> Epoch: 967, cost 14.63 s 2023-09-04 06:31:31,887 44k INFO ====> Epoch: 968, cost 14.21 s 2023-09-04 06:31:46,383 44k INFO ====> Epoch: 969, cost 14.50 s 2023-09-04 06:32:01,081 44k INFO ====> Epoch: 970, cost 14.70 s 2023-09-04 06:32:15,519 44k INFO ====> Epoch: 971, cost 14.44 s 2023-09-04 06:32:21,884 44k INFO Train Epoch: 972 [38%] 2023-09-04 06:32:21,884 44k INFO Losses: [2.030055046081543, 3.085148811340332, 7.617705821990967, 17.30803680419922, 0.6595214009284973], step: 20400, lr: 8.856950397368101e-05, reference_loss: 30.700468063354492 2023-09-04 06:32:30,326 44k INFO ====> Epoch: 972, cost 14.81 s 2023-09-04 06:32:44,820 44k INFO ====> Epoch: 973, cost 14.49 s 2023-09-04 06:32:59,473 44k INFO ====> Epoch: 974, cost 14.65 s 2023-09-04 06:33:13,887 44k INFO ====> Epoch: 975, cost 14.41 s 2023-09-04 06:33:28,277 44k INFO ====> Epoch: 976, cost 14.39 s 2023-09-04 06:33:42,620 44k INFO ====> Epoch: 977, cost 14.34 s 2023-09-04 06:33:57,437 44k INFO ====> Epoch: 978, cost 14.82 s 2023-09-04 06:34:11,877 44k INFO ====> Epoch: 979, cost 14.44 s 2023-09-04 06:34:26,180 44k INFO ====> Epoch: 980, cost 14.30 s 2023-09-04 06:34:40,322 44k INFO Train Epoch: 981 [90%] 2023-09-04 06:34:40,322 44k INFO Losses: [2.1987850666046143, 2.630373954772949, 8.409788131713867, 18.110748291015625, 0.5601584911346436], step: 20600, lr: 8.846991308752837e-05, reference_loss: 31.909852981567383 2023-09-04 06:34:41,091 44k INFO ====> Epoch: 981, cost 14.91 s 2023-09-04 06:34:55,803 44k INFO ====> Epoch: 982, cost 14.71 s 2023-09-04 06:35:10,334 44k INFO ====> Epoch: 983, cost 14.53 s 2023-09-04 06:35:24,551 44k INFO ====> Epoch: 984, cost 14.22 s 2023-09-04 06:35:38,857 44k INFO ====> Epoch: 985, cost 14.31 s 2023-09-04 06:35:53,684 44k INFO ====> Epoch: 986, cost 14.83 s 2023-09-04 06:36:08,662 44k INFO ====> Epoch: 987, cost 14.98 s 2023-09-04 06:36:23,039 44k INFO ====> Epoch: 988, cost 14.38 s 2023-09-04 06:36:37,630 44k INFO ====> Epoch: 989, cost 14.59 s 2023-09-04 06:36:52,202 44k INFO ====> Epoch: 990, cost 14.57 s 2023-09-04 06:36:59,500 44k INFO Train Epoch: 991 [43%] 2023-09-04 06:36:59,501 44k INFO Losses: [2.3310179710388184, 2.4348268508911133, 7.042314052581787, 15.324657440185547, 0.8069932460784912], step: 20800, lr: 8.835938788084596e-05, reference_loss: 27.939809799194336 2023-09-04 06:37:04,862 44k INFO Saving model and optimizer state at iteration 991 to ./logs\44k\G_20800.pth 2023-09-04 06:37:05,435 44k INFO Saving model and optimizer state at iteration 991 to ./logs\44k\D_20800.pth 2023-09-04 06:37:06,599 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_18400.pth 2023-09-04 06:37:06,636 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_18400.pth 2023-09-04 06:37:14,004 44k INFO ====> Epoch: 991, cost 21.80 s 2023-09-04 06:37:28,195 44k INFO ====> Epoch: 992, cost 14.19 s 2023-09-04 06:37:42,577 44k INFO ====> Epoch: 993, cost 14.38 s 2023-09-04 06:37:57,287 44k INFO ====> Epoch: 994, cost 14.71 s 2023-09-04 06:38:11,951 44k INFO ====> Epoch: 995, cost 14.66 s 2023-09-04 06:38:26,357 44k INFO ====> Epoch: 996, cost 14.41 s 2023-09-04 06:38:40,767 44k INFO ====> Epoch: 997, cost 14.41 s 2023-09-04 06:38:55,352 44k INFO ====> Epoch: 998, cost 14.59 s 2023-09-04 06:39:10,006 44k INFO ====> Epoch: 999, cost 14.65 s 2023-09-04 06:39:24,096 44k INFO Train Epoch: 1000 [95%] 2023-09-04 06:39:24,096 44k INFO Losses: [2.8671693801879883, 2.3667149543762207, 5.8498005867004395, 16.694597244262695, 0.7155517935752869], step: 21000, lr: 8.82600332571419e-05, reference_loss: 28.493833541870117 2023-09-04 06:39:24,360 44k INFO ====> Epoch: 1000, cost 14.35 s 2023-09-04 06:39:38,578 44k INFO ====> Epoch: 1001, cost 14.22 s 2023-09-04 06:39:53,078 44k INFO ====> Epoch: 1002, cost 14.50 s 2023-09-04 06:40:07,637 44k INFO ====> Epoch: 1003, cost 14.56 s 2023-09-04 06:40:22,111 44k INFO ====> Epoch: 1004, cost 14.47 s 2023-09-04 06:40:36,572 44k INFO ====> Epoch: 1005, cost 14.46 s 2023-09-04 06:40:51,233 44k INFO ====> Epoch: 1006, cost 14.66 s 2023-09-04 06:41:05,878 44k INFO ====> Epoch: 1007, cost 14.65 s 2023-09-04 06:41:20,164 44k INFO ====> Epoch: 1008, cost 14.29 s 2023-09-04 06:41:34,428 44k INFO ====> Epoch: 1009, cost 14.26 s 2023-09-04 06:41:42,155 44k INFO Train Epoch: 1010 [48%] 2023-09-04 06:41:42,155 44k INFO Losses: [2.423478841781616, 2.45825457572937, 8.206273078918457, 17.711637496948242, 0.7230669260025024], step: 21200, lr: 8.814977025272491e-05, reference_loss: 31.5227108001709 2023-09-04 06:41:49,828 44k INFO ====> Epoch: 1010, cost 15.40 s 2023-09-04 06:42:04,629 44k INFO ====> Epoch: 1011, cost 14.80 s 2023-09-04 06:42:19,263 44k INFO ====> Epoch: 1012, cost 14.63 s 2023-09-04 06:42:33,790 44k INFO ====> Epoch: 1013, cost 14.53 s 2023-09-04 06:42:48,281 44k INFO ====> Epoch: 1014, cost 14.49 s 2023-09-04 06:43:02,718 44k INFO ====> Epoch: 1015, cost 14.44 s 2023-09-04 06:43:17,016 44k INFO ====> Epoch: 1016, cost 14.30 s 2023-09-04 06:43:31,520 44k INFO ====> Epoch: 1017, cost 14.50 s 2023-09-04 06:43:45,922 44k INFO ====> Epoch: 1018, cost 14.40 s 2023-09-04 06:44:00,791 44k INFO ====> Epoch: 1019, cost 14.87 s 2023-09-04 06:44:01,521 44k INFO Train Epoch: 1020 [0%] 2023-09-04 06:44:01,522 44k INFO Losses: [2.590294122695923, 2.3002002239227295, 5.642866134643555, 14.645483016967773, 0.8113415837287903], step: 21400, lr: 8.803964499956059e-05, reference_loss: 25.99018669128418 2023-09-04 06:44:15,690 44k INFO ====> Epoch: 1020, cost 14.90 s 2023-09-04 06:44:30,097 44k INFO ====> Epoch: 1021, cost 14.41 s 2023-09-04 06:44:44,583 44k INFO ====> Epoch: 1022, cost 14.49 s 2023-09-04 06:44:59,140 44k INFO ====> Epoch: 1023, cost 14.56 s 2023-09-04 06:45:13,649 44k INFO ====> Epoch: 1024, cost 14.51 s 2023-09-04 06:45:28,111 44k INFO ====> Epoch: 1025, cost 14.46 s 2023-09-04 06:45:42,731 44k INFO ====> Epoch: 1026, cost 14.62 s 2023-09-04 06:45:57,380 44k INFO ====> Epoch: 1027, cost 14.65 s 2023-09-04 06:46:11,948 44k INFO ====> Epoch: 1028, cost 14.57 s 2023-09-04 06:46:20,502 44k INFO Train Epoch: 1029 [52%] 2023-09-04 06:46:20,503 44k INFO Losses: [2.2680037021636963, 2.6625113487243652, 7.794908046722412, 17.335063934326172, 0.7142598032951355], step: 21600, lr: 8.794064990679505e-05, reference_loss: 30.774747848510742 2023-09-04 06:46:26,056 44k INFO Saving model and optimizer state at iteration 1029 to ./logs\44k\G_21600.pth 2023-09-04 06:46:26,659 44k INFO Saving model and optimizer state at iteration 1029 to ./logs\44k\D_21600.pth 2023-09-04 06:46:27,896 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_19200.pth 2023-09-04 06:46:27,931 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_19200.pth 2023-09-04 06:46:34,070 44k INFO ====> Epoch: 1029, cost 22.12 s 2023-09-04 06:46:48,472 44k INFO ====> Epoch: 1030, cost 14.40 s 2023-09-04 06:47:03,111 44k INFO ====> Epoch: 1031, cost 14.64 s 2023-09-04 06:47:17,653 44k INFO ====> Epoch: 1032, cost 14.54 s 2023-09-04 06:47:32,228 44k INFO ====> Epoch: 1033, cost 14.57 s 2023-09-04 06:47:46,838 44k INFO ====> Epoch: 1034, cost 14.61 s 2023-09-04 06:48:01,572 44k INFO ====> Epoch: 1035, cost 14.73 s 2023-09-04 06:48:16,358 44k INFO ====> Epoch: 1036, cost 14.79 s 2023-09-04 06:48:30,789 44k INFO ====> Epoch: 1037, cost 14.43 s 2023-09-04 06:48:45,209 44k INFO ====> Epoch: 1038, cost 14.42 s 2023-09-04 06:48:46,628 44k INFO Train Epoch: 1039 [5%] 2023-09-04 06:48:46,628 44k INFO Losses: [2.3148272037506104, 2.746319532394409, 6.432289123535156, 15.150300979614258, 0.7878642678260803], step: 21800, lr: 8.783078590707442e-05, reference_loss: 27.431602478027344 2023-09-04 06:48:59,868 44k INFO ====> Epoch: 1039, cost 14.66 s 2023-09-04 06:49:14,393 44k INFO ====> Epoch: 1040, cost 14.53 s 2023-09-04 06:49:28,894 44k INFO ====> Epoch: 1041, cost 14.50 s 2023-09-04 06:49:43,510 44k INFO ====> Epoch: 1042, cost 14.62 s 2023-09-04 06:49:58,176 44k INFO ====> Epoch: 1043, cost 14.67 s 2023-09-04 06:50:12,745 44k INFO ====> Epoch: 1044, cost 14.57 s 2023-09-04 06:50:27,180 44k INFO ====> Epoch: 1045, cost 14.44 s 2023-09-04 06:50:41,534 44k INFO ====> Epoch: 1046, cost 14.35 s 2023-09-04 06:50:55,927 44k INFO ====> Epoch: 1047, cost 14.39 s 2023-09-04 06:51:04,946 44k INFO Train Epoch: 1048 [57%] 2023-09-04 06:51:04,947 44k INFO Losses: [2.3264081478118896, 2.507721424102783, 8.09449577331543, 18.29441261291504, 0.6254550814628601], step: 22000, lr: 8.773202566333896e-05, reference_loss: 31.848493576049805 2023-09-04 06:51:10,505 44k INFO ====> Epoch: 1048, cost 14.58 s 2023-09-04 06:51:25,000 44k INFO ====> Epoch: 1049, cost 14.50 s 2023-09-04 06:51:39,386 44k INFO ====> Epoch: 1050, cost 14.39 s 2023-09-04 06:51:54,107 44k INFO ====> Epoch: 1051, cost 14.72 s 2023-09-04 06:52:08,776 44k INFO ====> Epoch: 1052, cost 14.67 s 2023-09-04 06:52:23,408 44k INFO ====> Epoch: 1053, cost 14.63 s 2023-09-04 06:52:37,797 44k INFO ====> Epoch: 1054, cost 14.39 s 2023-09-04 06:52:52,455 44k INFO ====> Epoch: 1055, cost 14.66 s 2023-09-04 06:53:07,217 44k INFO ====> Epoch: 1056, cost 14.76 s 2023-09-04 06:53:21,613 44k INFO ====> Epoch: 1057, cost 14.40 s 2023-09-04 06:53:23,678 44k INFO Train Epoch: 1058 [10%] 2023-09-04 06:53:23,679 44k INFO Losses: [2.6077027320861816, 2.200411081314087, 5.917771339416504, 16.084394454956055, 0.7312707901000977], step: 22200, lr: 8.76224222972826e-05, reference_loss: 27.541549682617188 2023-09-04 06:53:36,118 44k INFO ====> Epoch: 1058, cost 14.51 s 2023-09-04 06:53:50,998 44k INFO ====> Epoch: 1059, cost 14.88 s 2023-09-04 06:54:06,144 44k INFO ====> Epoch: 1060, cost 15.15 s 2023-09-04 06:54:20,746 44k INFO ====> Epoch: 1061, cost 14.60 s 2023-09-04 06:54:35,398 44k INFO ====> Epoch: 1062, cost 14.65 s 2023-09-04 06:54:50,310 44k INFO ====> Epoch: 1063, cost 14.91 s 2023-09-04 06:55:05,031 44k INFO ====> Epoch: 1064, cost 14.72 s 2023-09-04 06:55:19,426 44k INFO ====> Epoch: 1065, cost 14.40 s 2023-09-04 06:55:33,701 44k INFO ====> Epoch: 1066, cost 14.27 s 2023-09-04 06:55:43,625 44k INFO Train Epoch: 1067 [62%] 2023-09-04 06:55:43,626 44k INFO Losses: [2.3112945556640625, 2.3596179485321045, 6.059848785400391, 15.693866729736328, 0.7136959433555603], step: 22400, lr: 8.75238963454378e-05, reference_loss: 27.138324737548828 2023-09-04 06:55:49,002 44k INFO Saving model and optimizer state at iteration 1067 to ./logs\44k\G_22400.pth 2023-09-04 06:55:49,680 44k INFO Saving model and optimizer state at iteration 1067 to ./logs\44k\D_22400.pth 2023-09-04 06:55:50,799 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_20000.pth 2023-09-04 06:55:50,834 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_20000.pth 2023-09-04 06:55:55,507 44k INFO ====> Epoch: 1067, cost 21.81 s 2023-09-04 06:56:10,031 44k INFO ====> Epoch: 1068, cost 14.52 s 2023-09-04 06:56:24,565 44k INFO ====> Epoch: 1069, cost 14.53 s 2023-09-04 06:56:38,989 44k INFO ====> Epoch: 1070, cost 14.42 s 2023-09-04 06:56:53,657 44k INFO ====> Epoch: 1071, cost 14.67 s 2023-09-04 06:57:08,264 44k INFO ====> Epoch: 1072, cost 14.61 s 2023-09-04 06:57:22,562 44k INFO ====> Epoch: 1073, cost 14.30 s 2023-09-04 06:57:36,800 44k INFO ====> Epoch: 1074, cost 14.24 s 2023-09-04 06:57:51,386 44k INFO ====> Epoch: 1075, cost 14.59 s 2023-09-04 06:58:06,414 44k INFO ====> Epoch: 1076, cost 15.03 s 2023-09-04 06:58:09,296 44k INFO Train Epoch: 1077 [14%] 2023-09-04 06:58:09,296 44k INFO Losses: [2.3925046920776367, 2.6768627166748047, 7.722132682800293, 18.502201080322266, 0.8078364729881287], step: 22600, lr: 8.741455299473667e-05, reference_loss: 32.10153579711914 2023-09-04 06:58:21,276 44k INFO ====> Epoch: 1077, cost 14.86 s 2023-09-04 06:58:35,781 44k INFO ====> Epoch: 1078, cost 14.51 s 2023-09-04 06:58:50,524 44k INFO ====> Epoch: 1079, cost 14.74 s 2023-09-04 06:59:05,409 44k INFO ====> Epoch: 1080, cost 14.88 s 2023-09-04 06:59:20,106 44k INFO ====> Epoch: 1081, cost 14.70 s 2023-09-04 06:59:34,593 44k INFO ====> Epoch: 1082, cost 14.49 s 2023-09-04 06:59:49,325 44k INFO ====> Epoch: 1083, cost 14.73 s 2023-09-04 07:00:03,800 44k INFO ====> Epoch: 1084, cost 14.47 s 2023-09-04 07:00:18,461 44k INFO ====> Epoch: 1085, cost 14.66 s 2023-09-04 07:00:29,067 44k INFO Train Epoch: 1086 [67%] 2023-09-04 07:00:29,067 44k INFO Losses: [2.413604259490967, 2.4827473163604736, 7.166926860809326, 17.30841636657715, 0.6327067017555237], step: 22800, lr: 8.731626077896486e-05, reference_loss: 30.00440216064453 2023-09-04 07:00:33,305 44k INFO ====> Epoch: 1086, cost 14.84 s 2023-09-04 07:00:48,007 44k INFO ====> Epoch: 1087, cost 14.70 s 2023-09-04 07:01:02,507 44k INFO ====> Epoch: 1088, cost 14.50 s 2023-09-04 07:01:16,949 44k INFO ====> Epoch: 1089, cost 14.44 s 2023-09-04 07:01:31,479 44k INFO ====> Epoch: 1090, cost 14.53 s 2023-09-04 07:01:46,030 44k INFO ====> Epoch: 1091, cost 14.55 s 2023-09-04 07:02:00,822 44k INFO ====> Epoch: 1092, cost 14.79 s 2023-09-04 07:02:15,530 44k INFO ====> Epoch: 1093, cost 14.71 s 2023-09-04 07:02:30,139 44k INFO ====> Epoch: 1094, cost 14.61 s 2023-09-04 07:02:44,607 44k INFO ====> Epoch: 1095, cost 14.47 s 2023-09-04 07:02:48,070 44k INFO Train Epoch: 1096 [19%] 2023-09-04 07:02:48,070 44k INFO Losses: [2.127730369567871, 2.884443759918213, 8.158646583557129, 17.259170532226562, 0.6143352389335632], step: 23000, lr: 8.72071768267767e-05, reference_loss: 31.044326782226562 2023-09-04 07:02:59,445 44k INFO ====> Epoch: 1096, cost 14.84 s 2023-09-04 07:03:13,926 44k INFO ====> Epoch: 1097, cost 14.48 s 2023-09-04 07:03:28,578 44k INFO ====> Epoch: 1098, cost 14.65 s 2023-09-04 07:03:43,914 44k INFO ====> Epoch: 1099, cost 15.34 s 2023-09-04 07:03:58,648 44k INFO ====> Epoch: 1100, cost 14.73 s 2023-09-04 07:04:13,421 44k INFO ====> Epoch: 1101, cost 14.77 s 2023-09-04 07:04:27,877 44k INFO ====> Epoch: 1102, cost 14.46 s 2023-09-04 07:04:42,414 44k INFO ====> Epoch: 1103, cost 14.54 s 2023-09-04 07:04:57,069 44k INFO ====> Epoch: 1104, cost 14.65 s 2023-09-04 07:05:08,376 44k INFO Train Epoch: 1105 [71%] 2023-09-04 07:05:08,377 44k INFO Losses: [2.388706684112549, 2.4669578075408936, 7.462087631225586, 16.852062225341797, 0.7768691778182983], step: 23200, lr: 8.710911779257877e-05, reference_loss: 29.946683883666992 2023-09-04 07:05:13,882 44k INFO Saving model and optimizer state at iteration 1105 to ./logs\44k\G_23200.pth 2023-09-04 07:05:14,502 44k INFO Saving model and optimizer state at iteration 1105 to ./logs\44k\D_23200.pth 2023-09-04 07:05:15,636 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_20800.pth 2023-09-04 07:05:15,669 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_20800.pth 2023-09-04 07:05:18,811 44k INFO ====> Epoch: 1105, cost 21.74 s 2023-09-04 07:05:33,234 44k INFO ====> Epoch: 1106, cost 14.42 s 2023-09-04 07:05:47,862 44k INFO ====> Epoch: 1107, cost 14.63 s 2023-09-04 07:06:02,572 44k INFO ====> Epoch: 1108, cost 14.71 s 2023-09-04 07:06:17,097 44k INFO ====> Epoch: 1109, cost 14.53 s 2023-09-04 07:06:31,628 44k INFO ====> Epoch: 1110, cost 14.53 s 2023-09-04 07:06:46,140 44k INFO ====> Epoch: 1111, cost 14.51 s 2023-09-04 07:07:00,746 44k INFO ====> Epoch: 1112, cost 14.61 s 2023-09-04 07:07:15,151 44k INFO ====> Epoch: 1113, cost 14.41 s 2023-09-04 07:07:29,689 44k INFO ====> Epoch: 1114, cost 14.54 s 2023-09-04 07:07:33,850 44k INFO Train Epoch: 1115 [24%] 2023-09-04 07:07:33,850 44k INFO Losses: [2.468508720397949, 2.519362449645996, 6.245070934295654, 15.838983535766602, 0.7118121385574341], step: 23400, lr: 8.700029262352475e-05, reference_loss: 27.78373908996582 2023-09-04 07:07:44,315 44k INFO ====> Epoch: 1115, cost 14.63 s 2023-09-04 07:07:59,099 44k INFO ====> Epoch: 1116, cost 14.78 s 2023-09-04 07:08:13,619 44k INFO ====> Epoch: 1117, cost 14.52 s 2023-09-04 07:08:28,235 44k INFO ====> Epoch: 1118, cost 14.62 s 2023-09-04 07:08:42,616 44k INFO ====> Epoch: 1119, cost 14.38 s 2023-09-04 07:08:57,069 44k INFO ====> Epoch: 1120, cost 14.45 s 2023-09-04 07:09:11,665 44k INFO ====> Epoch: 1121, cost 14.60 s 2023-09-04 07:09:26,290 44k INFO ====> Epoch: 1122, cost 14.63 s 2023-09-04 07:09:40,703 44k INFO ====> Epoch: 1123, cost 14.41 s 2023-09-04 07:09:52,745 44k INFO Train Epoch: 1124 [76%] 2023-09-04 07:09:52,745 44k INFO Losses: [2.3952560424804688, 2.26279878616333, 6.977020263671875, 17.698684692382812, 0.6497833132743835], step: 23600, lr: 8.690246621771705e-05, reference_loss: 29.983543395996094 2023-09-04 07:09:55,541 44k INFO ====> Epoch: 1124, cost 14.84 s 2023-09-04 07:10:10,115 44k INFO ====> Epoch: 1125, cost 14.57 s 2023-09-04 07:10:24,757 44k INFO ====> Epoch: 1126, cost 14.64 s 2023-09-04 07:10:39,301 44k INFO ====> Epoch: 1127, cost 14.54 s 2023-09-04 07:10:53,996 44k INFO ====> Epoch: 1128, cost 14.70 s 2023-09-04 07:11:08,897 44k INFO ====> Epoch: 1129, cost 14.90 s 2023-09-04 07:11:23,201 44k INFO ====> Epoch: 1130, cost 14.30 s 2023-09-04 07:11:37,572 44k INFO ====> Epoch: 1131, cost 14.37 s 2023-09-04 07:11:52,547 44k INFO ====> Epoch: 1132, cost 14.97 s 2023-09-04 07:12:07,333 44k INFO ====> Epoch: 1133, cost 14.79 s 2023-09-04 07:12:12,270 44k INFO Train Epoch: 1134 [29%] 2023-09-04 07:12:12,271 44k INFO Losses: [2.4043214321136475, 2.2563910484313965, 7.0517377853393555, 14.27387523651123, 0.6044024229049683], step: 23800, lr: 8.679389921787813e-05, reference_loss: 26.590726852416992 2023-09-04 07:12:22,019 44k INFO ====> Epoch: 1134, cost 14.69 s 2023-09-04 07:12:36,457 44k INFO ====> Epoch: 1135, cost 14.44 s 2023-09-04 07:12:51,104 44k INFO ====> Epoch: 1136, cost 14.65 s 2023-09-04 07:13:05,869 44k INFO ====> Epoch: 1137, cost 14.77 s 2023-09-04 07:13:20,400 44k INFO ====> Epoch: 1138, cost 14.53 s 2023-09-04 07:13:34,646 44k INFO ====> Epoch: 1139, cost 14.25 s 2023-09-04 07:13:49,321 44k INFO ====> Epoch: 1140, cost 14.67 s 2023-09-04 07:14:04,197 44k INFO ====> Epoch: 1141, cost 14.88 s 2023-09-04 07:14:18,620 44k INFO ====> Epoch: 1142, cost 14.42 s 2023-09-04 07:14:31,281 44k INFO Train Epoch: 1143 [81%] 2023-09-04 07:14:31,282 44k INFO Losses: [2.0903549194335938, 2.68072509765625, 6.580922603607178, 14.923503875732422, 0.7251425385475159], step: 24000, lr: 8.669630488858935e-05, reference_loss: 27.00065040588379 2023-09-04 07:14:36,723 44k INFO Saving model and optimizer state at iteration 1143 to ./logs\44k\G_24000.pth 2023-09-04 07:14:37,342 44k INFO Saving model and optimizer state at iteration 1143 to ./logs\44k\D_24000.pth 2023-09-04 07:14:38,522 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_21600.pth 2023-09-04 07:14:38,558 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_21600.pth 2023-09-04 07:14:40,485 44k INFO ====> Epoch: 1143, cost 21.87 s 2023-09-04 07:14:54,910 44k INFO ====> Epoch: 1144, cost 14.42 s 2023-09-04 07:15:09,435 44k INFO ====> Epoch: 1145, cost 14.53 s 2023-09-04 07:15:23,652 44k INFO ====> Epoch: 1146, cost 14.22 s 2023-09-04 07:15:38,056 44k INFO ====> Epoch: 1147, cost 14.40 s 2023-09-04 07:15:52,568 44k INFO ====> Epoch: 1148, cost 14.51 s 2023-09-04 07:16:07,108 44k INFO ====> Epoch: 1149, cost 14.54 s 2023-09-04 07:16:21,601 44k INFO ====> Epoch: 1150, cost 14.49 s 2023-09-04 07:16:36,048 44k INFO ====> Epoch: 1151, cost 14.45 s 2023-09-04 07:16:50,422 44k INFO ====> Epoch: 1152, cost 14.37 s 2023-09-04 07:16:56,146 44k INFO Train Epoch: 1153 [33%] 2023-09-04 07:16:56,147 44k INFO Losses: [2.4429128170013428, 2.589897632598877, 7.138553619384766, 15.340700149536133, 0.6626195311546326], step: 24200, lr: 8.658799544550295e-05, reference_loss: 28.1746826171875 2023-09-04 07:17:05,481 44k INFO ====> Epoch: 1153, cost 15.06 s 2023-09-04 07:17:19,979 44k INFO ====> Epoch: 1154, cost 14.50 s 2023-09-04 07:17:34,185 44k INFO ====> Epoch: 1155, cost 14.21 s 2023-09-04 07:17:48,914 44k INFO ====> Epoch: 1156, cost 14.73 s 2023-09-04 07:18:03,696 44k INFO ====> Epoch: 1157, cost 14.78 s 2023-09-04 07:18:18,380 44k INFO ====> Epoch: 1158, cost 14.68 s 2023-09-04 07:18:32,819 44k INFO ====> Epoch: 1159, cost 14.44 s 2023-09-04 07:18:47,595 44k INFO ====> Epoch: 1160, cost 14.78 s 2023-09-04 07:19:02,215 44k INFO ====> Epoch: 1161, cost 14.62 s 2023-09-04 07:19:15,649 44k INFO Train Epoch: 1162 [86%] 2023-09-04 07:19:15,650 44k INFO Losses: [2.3678839206695557, 2.3701932430267334, 6.88004207611084, 16.735233306884766, 0.6371505856513977], step: 24400, lr: 8.649063264217098e-05, reference_loss: 28.990501403808594 2023-09-04 07:19:16,990 44k INFO ====> Epoch: 1162, cost 14.78 s 2023-09-04 07:19:31,144 44k INFO ====> Epoch: 1163, cost 14.15 s 2023-09-04 07:19:45,721 44k INFO ====> Epoch: 1164, cost 14.58 s 2023-09-04 07:20:00,731 44k INFO ====> Epoch: 1165, cost 15.01 s 2023-09-04 07:20:15,341 44k INFO ====> Epoch: 1166, cost 14.61 s 2023-09-04 07:20:29,658 44k INFO ====> Epoch: 1167, cost 14.32 s 2023-09-04 07:20:44,219 44k INFO ====> Epoch: 1168, cost 14.56 s 2023-09-04 07:20:58,598 44k INFO ====> Epoch: 1169, cost 14.38 s 2023-09-04 07:21:12,959 44k INFO ====> Epoch: 1170, cost 14.36 s 2023-09-04 07:21:27,350 44k INFO ====> Epoch: 1171, cost 14.39 s 2023-09-04 07:21:33,736 44k INFO Train Epoch: 1172 [38%] 2023-09-04 07:21:33,736 44k INFO Losses: [2.561049222946167, 2.5585885047912598, 6.1490888595581055, 16.814014434814453, 0.5132483839988708], step: 24600, lr: 8.638258014482751e-05, reference_loss: 28.595989227294922 2023-09-04 07:21:42,225 44k INFO ====> Epoch: 1172, cost 14.87 s 2023-09-04 07:21:56,681 44k INFO ====> Epoch: 1173, cost 14.46 s 2023-09-04 07:22:11,391 44k INFO ====> Epoch: 1174, cost 14.71 s 2023-09-04 07:22:25,912 44k INFO ====> Epoch: 1175, cost 14.52 s 2023-09-04 07:22:40,301 44k INFO ====> Epoch: 1176, cost 14.39 s 2023-09-04 07:22:54,948 44k INFO ====> Epoch: 1177, cost 14.65 s 2023-09-04 07:23:09,694 44k INFO ====> Epoch: 1178, cost 14.75 s 2023-09-04 07:23:24,006 44k INFO ====> Epoch: 1179, cost 14.31 s 2023-09-04 07:23:38,411 44k INFO ====> Epoch: 1180, cost 14.40 s 2023-09-04 07:23:52,589 44k INFO Train Epoch: 1181 [90%] 2023-09-04 07:23:52,589 44k INFO Losses: [2.470487117767334, 2.357884168624878, 6.056527137756348, 16.079011917114258, 0.8033779263496399], step: 24800, lr: 8.628544831819641e-05, reference_loss: 27.76728630065918 2023-09-04 07:23:58,202 44k INFO Saving model and optimizer state at iteration 1181 to ./logs\44k\G_24800.pth 2023-09-04 07:23:58,771 44k INFO Saving model and optimizer state at iteration 1181 to ./logs\44k\D_24800.pth 2023-09-04 07:24:00,468 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_22400.pth 2023-09-04 07:24:00,503 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_22400.pth 2023-09-04 07:24:00,965 44k INFO ====> Epoch: 1181, cost 22.55 s 2023-09-04 07:24:15,527 44k INFO ====> Epoch: 1182, cost 14.56 s 2023-09-04 07:24:29,782 44k INFO ====> Epoch: 1183, cost 14.26 s 2023-09-04 07:24:44,259 44k INFO ====> Epoch: 1184, cost 14.48 s 2023-09-04 07:24:58,736 44k INFO ====> Epoch: 1185, cost 14.48 s 2023-09-04 07:25:13,007 44k INFO ====> Epoch: 1186, cost 14.27 s 2023-09-04 07:25:27,098 44k INFO ====> Epoch: 1187, cost 14.09 s 2023-09-04 07:25:41,189 44k INFO ====> Epoch: 1188, cost 14.09 s 2023-09-04 07:25:55,744 44k INFO ====> Epoch: 1189, cost 14.56 s 2023-09-04 07:26:10,090 44k INFO ====> Epoch: 1190, cost 14.35 s 2023-09-04 07:26:16,998 44k INFO Train Epoch: 1191 [43%] 2023-09-04 07:26:16,998 44k INFO Losses: [2.4484407901763916, 2.3458621501922607, 7.134237289428711, 16.93844985961914, 0.6454229354858398], step: 25000, lr: 8.617765215703574e-05, reference_loss: 29.512413024902344 2023-09-04 07:26:24,644 44k INFO ====> Epoch: 1191, cost 14.55 s 2023-09-04 07:26:38,940 44k INFO ====> Epoch: 1192, cost 14.30 s 2023-09-04 07:26:53,199 44k INFO ====> Epoch: 1193, cost 14.26 s 2023-09-04 07:27:07,712 44k INFO ====> Epoch: 1194, cost 14.51 s 2023-09-04 07:27:21,815 44k INFO ====> Epoch: 1195, cost 14.10 s 2023-09-04 07:27:35,879 44k INFO ====> Epoch: 1196, cost 14.06 s 2023-09-04 07:27:50,083 44k INFO ====> Epoch: 1197, cost 14.20 s 2023-09-04 07:28:04,613 44k INFO ====> Epoch: 1198, cost 14.53 s 2023-09-04 07:28:18,813 44k INFO ====> Epoch: 1199, cost 14.20 s 2023-09-04 07:28:32,844 44k INFO Train Epoch: 1200 [95%] 2023-09-04 07:28:32,844 44k INFO Losses: [2.2562711238861084, 2.8789784908294678, 8.9086275100708, 18.102191925048828, 0.3074663281440735], step: 25200, lr: 8.608075075915251e-05, reference_loss: 32.45353317260742 2023-09-04 07:28:33,101 44k INFO ====> Epoch: 1200, cost 14.29 s 2023-09-04 07:28:47,443 44k INFO ====> Epoch: 1201, cost 14.34 s 2023-09-04 07:29:01,878 44k INFO ====> Epoch: 1202, cost 14.44 s 2023-09-04 07:29:16,082 44k INFO ====> Epoch: 1203, cost 14.20 s 2023-09-04 07:29:30,230 44k INFO ====> Epoch: 1204, cost 14.15 s 2023-09-04 07:29:44,460 44k INFO ====> Epoch: 1205, cost 14.23 s 2023-09-04 07:29:59,231 44k INFO ====> Epoch: 1206, cost 14.77 s 2023-09-04 07:30:13,573 44k INFO ====> Epoch: 1207, cost 14.34 s 2023-09-04 07:30:27,592 44k INFO ====> Epoch: 1208, cost 14.02 s 2023-09-04 07:30:41,692 44k INFO ====> Epoch: 1209, cost 14.10 s 2023-09-04 07:30:49,259 44k INFO Train Epoch: 1210 [48%] 2023-09-04 07:30:49,259 44k INFO Losses: [2.478944778442383, 2.3251771926879883, 7.360312461853027, 16.926877975463867, 0.6325295567512512], step: 25400, lr: 8.597321032606063e-05, reference_loss: 29.72384262084961 2023-09-04 07:30:56,220 44k INFO ====> Epoch: 1210, cost 14.53 s 2023-09-04 07:31:10,452 44k INFO ====> Epoch: 1211, cost 14.23 s 2023-09-04 07:31:24,413 44k INFO ====> Epoch: 1212, cost 13.96 s 2023-09-04 07:31:38,334 44k INFO ====> Epoch: 1213, cost 13.92 s 2023-09-04 07:31:52,581 44k INFO ====> Epoch: 1214, cost 14.25 s 2023-09-04 07:32:06,854 44k INFO ====> Epoch: 1215, cost 14.27 s 2023-09-04 07:32:21,108 44k INFO ====> Epoch: 1216, cost 14.25 s 2023-09-04 07:32:35,099 44k INFO ====> Epoch: 1217, cost 13.99 s 2023-09-04 07:32:49,272 44k INFO ====> Epoch: 1218, cost 14.17 s 2023-09-04 07:33:03,620 44k INFO ====> Epoch: 1219, cost 14.35 s 2023-09-04 07:33:04,297 44k INFO Train Epoch: 1220 [0%] 2023-09-04 07:33:04,298 44k INFO Losses: [2.359585762023926, 2.6116695404052734, 6.7839035987854, 16.226415634155273, 0.6804808378219604], step: 25600, lr: 8.586580424292098e-05, reference_loss: 28.66205596923828 2023-09-04 07:33:08,948 44k INFO Saving model and optimizer state at iteration 1220 to ./logs\44k\G_25600.pth 2023-09-04 07:33:09,562 44k INFO Saving model and optimizer state at iteration 1220 to ./logs\44k\D_25600.pth 2023-09-04 07:33:10,662 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_23200.pth 2023-09-04 07:33:10,696 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_23200.pth 2023-09-04 07:33:24,009 44k INFO ====> Epoch: 1220, cost 20.39 s 2023-09-04 07:33:37,967 44k INFO ====> Epoch: 1221, cost 13.96 s 2023-09-04 07:33:52,250 44k INFO ====> Epoch: 1222, cost 14.28 s 2023-09-04 07:34:06,462 44k INFO ====> Epoch: 1223, cost 14.21 s 2023-09-04 07:34:20,491 44k INFO ====> Epoch: 1224, cost 14.03 s 2023-09-04 07:34:34,437 44k INFO ====> Epoch: 1225, cost 13.95 s 2023-09-04 07:34:48,727 44k INFO ====> Epoch: 1226, cost 14.29 s 2023-09-04 07:35:02,806 44k INFO ====> Epoch: 1227, cost 14.08 s 2023-09-04 07:35:16,998 44k INFO ====> Epoch: 1228, cost 14.19 s 2023-09-04 07:35:25,094 44k INFO Train Epoch: 1229 [52%] 2023-09-04 07:35:25,095 44k INFO Losses: [2.455819606781006, 2.271712064743042, 6.2211737632751465, 16.453821182250977, 0.7192624807357788], step: 25800, lr: 8.576925349857781e-05, reference_loss: 28.121789932250977 2023-09-04 07:35:31,188 44k INFO ====> Epoch: 1229, cost 14.19 s 2023-09-04 07:35:45,272 44k INFO ====> Epoch: 1230, cost 14.08 s 2023-09-04 07:35:59,598 44k INFO ====> Epoch: 1231, cost 14.33 s 2023-09-04 07:36:13,784 44k INFO ====> Epoch: 1232, cost 14.19 s 2023-09-04 07:36:27,882 44k INFO ====> Epoch: 1233, cost 14.10 s 2023-09-04 07:36:41,938 44k INFO ====> Epoch: 1234, cost 14.06 s 2023-09-04 07:36:56,172 44k INFO ====> Epoch: 1235, cost 14.23 s 2023-09-04 07:37:10,395 44k INFO ====> Epoch: 1236, cost 14.22 s 2023-09-04 07:37:24,405 44k INFO ====> Epoch: 1237, cost 14.01 s 2023-09-04 07:37:38,357 44k INFO ====> Epoch: 1238, cost 13.95 s 2023-09-04 07:37:39,749 44k INFO Train Epoch: 1239 [5%] 2023-09-04 07:37:39,749 44k INFO Losses: [2.429551839828491, 2.444643020629883, 7.2991042137146, 16.276649475097656, 0.5622893571853638], step: 26000, lr: 8.566210221811315e-05, reference_loss: 29.012239456176758 2023-09-04 07:37:52,759 44k INFO ====> Epoch: 1239, cost 14.40 s 2023-09-04 07:38:07,176 44k INFO ====> Epoch: 1240, cost 14.42 s 2023-09-04 07:38:21,234 44k INFO ====> Epoch: 1241, cost 14.06 s 2023-09-04 07:38:35,123 44k INFO ====> Epoch: 1242, cost 13.89 s 2023-09-04 07:38:49,289 44k INFO ====> Epoch: 1243, cost 14.17 s 2023-09-04 07:39:03,498 44k INFO ====> Epoch: 1244, cost 14.21 s 2023-09-04 07:39:17,453 44k INFO ====> Epoch: 1245, cost 13.96 s 2023-09-04 07:39:31,407 44k INFO ====> Epoch: 1246, cost 13.95 s 2023-09-04 07:39:45,494 44k INFO ====> Epoch: 1247, cost 14.09 s 2023-09-04 07:39:54,383 44k INFO Train Epoch: 1248 [57%] 2023-09-04 07:39:54,383 44k INFO Losses: [2.4202775955200195, 2.336665630340576, 6.627160549163818, 16.6260986328125, 0.6055824160575867], step: 26200, lr: 8.556578052399892e-05, reference_loss: 28.615785598754883 2023-09-04 07:39:59,884 44k INFO ====> Epoch: 1248, cost 14.39 s 2023-09-04 07:40:13,889 44k INFO ====> Epoch: 1249, cost 14.00 s 2023-09-04 07:40:27,954 44k INFO ====> Epoch: 1250, cost 14.07 s 2023-09-04 07:40:42,003 44k INFO ====> Epoch: 1251, cost 14.05 s 2023-09-04 07:40:56,262 44k INFO ====> Epoch: 1252, cost 14.26 s 2023-09-04 07:41:10,382 44k INFO ====> Epoch: 1253, cost 14.12 s 2023-09-04 07:41:24,505 44k INFO ====> Epoch: 1254, cost 14.12 s 2023-09-04 07:41:38,547 44k INFO ====> Epoch: 1255, cost 14.04 s 2023-09-04 07:41:52,865 44k INFO ====> Epoch: 1256, cost 14.32 s 2023-09-04 07:42:07,183 44k INFO ====> Epoch: 1257, cost 14.32 s 2023-09-04 07:42:09,257 44k INFO Train Epoch: 1258 [10%] 2023-09-04 07:42:09,257 44k INFO Losses: [2.6189653873443604, 2.2470977306365967, 7.359108924865723, 14.895164489746094, 0.814770519733429], step: 26400, lr: 8.545888344173321e-05, reference_loss: 27.93510627746582 2023-09-04 07:42:13,893 44k INFO Saving model and optimizer state at iteration 1258 to ./logs\44k\G_26400.pth 2023-09-04 07:42:14,495 44k INFO Saving model and optimizer state at iteration 1258 to ./logs\44k\D_26400.pth 2023-09-04 07:42:16,188 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_24000.pth 2023-09-04 07:42:16,222 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_24000.pth 2023-09-04 07:42:28,169 44k INFO ====> Epoch: 1258, cost 20.99 s 2023-09-04 07:42:42,341 44k INFO ====> Epoch: 1259, cost 14.17 s 2023-09-04 07:42:56,711 44k INFO ====> Epoch: 1260, cost 14.37 s 2023-09-04 07:43:11,103 44k INFO ====> Epoch: 1261, cost 14.39 s 2023-09-04 07:43:24,968 44k INFO ====> Epoch: 1262, cost 13.87 s 2023-09-04 07:43:39,064 44k INFO ====> Epoch: 1263, cost 14.10 s 2023-09-04 07:43:53,242 44k INFO ====> Epoch: 1264, cost 14.18 s 2023-09-04 07:44:07,690 44k INFO ====> Epoch: 1265, cost 14.45 s 2023-09-04 07:44:21,890 44k INFO ====> Epoch: 1266, cost 14.20 s 2023-09-04 07:44:31,495 44k INFO Train Epoch: 1267 [62%] 2023-09-04 07:44:31,495 44k INFO Losses: [2.3881900310516357, 2.416823387145996, 6.574004650115967, 17.738811492919922, 0.5839664936065674], step: 26600, lr: 8.536279025446519e-05, reference_loss: 29.701797485351562 2023-09-04 07:44:36,232 44k INFO ====> Epoch: 1267, cost 14.34 s 2023-09-04 07:44:50,417 44k INFO ====> Epoch: 1268, cost 14.19 s 2023-09-04 07:45:04,633 44k INFO ====> Epoch: 1269, cost 14.22 s 2023-09-04 07:45:18,727 44k INFO ====> Epoch: 1270, cost 14.09 s 2023-09-04 07:45:32,698 44k INFO ====> Epoch: 1271, cost 13.97 s 2023-09-04 07:45:46,827 44k INFO ====> Epoch: 1272, cost 14.13 s 2023-09-04 07:46:01,107 44k INFO ====> Epoch: 1273, cost 14.28 s 2023-09-04 07:46:15,209 44k INFO ====> Epoch: 1274, cost 14.10 s 2023-09-04 07:46:29,150 44k INFO ====> Epoch: 1275, cost 13.94 s 2023-09-04 07:46:43,238 44k INFO ====> Epoch: 1276, cost 14.09 s 2023-09-04 07:46:46,059 44k INFO Train Epoch: 1277 [14%] 2023-09-04 07:46:46,059 44k INFO Losses: [2.281118869781494, 2.46766996383667, 7.169788837432861, 17.324695587158203, 0.6463931798934937], step: 26800, lr: 8.525614676735643e-05, reference_loss: 29.889665603637695 2023-09-04 07:46:57,917 44k INFO ====> Epoch: 1277, cost 14.68 s 2023-09-04 07:47:11,995 44k INFO ====> Epoch: 1278, cost 14.08 s 2023-09-04 07:47:25,807 44k INFO ====> Epoch: 1279, cost 13.81 s 2023-09-04 07:47:39,934 44k INFO ====> Epoch: 1280, cost 14.13 s 2023-09-04 07:47:54,400 44k INFO ====> Epoch: 1281, cost 14.47 s 2023-09-04 07:48:08,747 44k INFO ====> Epoch: 1282, cost 14.35 s 2023-09-04 07:48:22,762 44k INFO ====> Epoch: 1283, cost 14.01 s 2023-09-04 07:48:37,071 44k INFO ====> Epoch: 1284, cost 14.31 s 2023-09-04 07:48:51,429 44k INFO ====> Epoch: 1285, cost 14.36 s 2023-09-04 07:49:01,757 44k INFO Train Epoch: 1286 [67%] 2023-09-04 07:49:01,757 44k INFO Losses: [2.300126314163208, 2.6766891479492188, 6.723458290100098, 14.451241493225098, 0.5629031658172607], step: 27000, lr: 8.516028154484098e-05, reference_loss: 26.714418411254883 2023-09-04 07:49:05,843 44k INFO ====> Epoch: 1286, cost 14.41 s 2023-09-04 07:49:19,906 44k INFO ====> Epoch: 1287, cost 14.06 s 2023-09-04 07:49:33,985 44k INFO ====> Epoch: 1288, cost 14.08 s 2023-09-04 07:49:48,016 44k INFO ====> Epoch: 1289, cost 14.03 s 2023-09-04 07:50:02,261 44k INFO ====> Epoch: 1290, cost 14.25 s 2023-09-04 07:50:16,713 44k INFO ====> Epoch: 1291, cost 14.45 s 2023-09-04 07:50:30,785 44k INFO ====> Epoch: 1292, cost 14.07 s 2023-09-04 07:50:45,021 44k INFO ====> Epoch: 1293, cost 14.24 s 2023-09-04 07:50:59,409 44k INFO ====> Epoch: 1294, cost 14.39 s 2023-09-04 07:51:13,505 44k INFO ====> Epoch: 1295, cost 14.10 s 2023-09-04 07:51:16,869 44k INFO Train Epoch: 1296 [19%] 2023-09-04 07:51:16,869 44k INFO Losses: [2.581838369369507, 2.3389387130737305, 6.95993709564209, 15.606596946716309, 0.72002112865448], step: 27200, lr: 8.505389105127777e-05, reference_loss: 28.207332611083984 2023-09-04 07:51:21,499 44k INFO Saving model and optimizer state at iteration 1296 to ./logs\44k\G_27200.pth 2023-09-04 07:51:22,068 44k INFO Saving model and optimizer state at iteration 1296 to ./logs\44k\D_27200.pth 2023-09-04 07:51:23,215 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_24800.pth 2023-09-04 07:51:23,248 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_24800.pth 2023-09-04 07:51:33,809 44k INFO ====> Epoch: 1296, cost 20.30 s 2023-09-04 07:51:47,892 44k INFO ====> Epoch: 1297, cost 14.08 s 2023-09-04 07:52:02,083 44k INFO ====> Epoch: 1298, cost 14.19 s 2023-09-04 07:52:16,225 44k INFO ====> Epoch: 1299, cost 14.14 s 2023-09-04 07:52:30,413 44k INFO ====> Epoch: 1300, cost 14.19 s 2023-09-04 07:52:44,618 44k INFO ====> Epoch: 1301, cost 14.21 s 2023-09-04 07:52:58,918 44k INFO ====> Epoch: 1302, cost 14.30 s 2023-09-04 07:53:13,217 44k INFO ====> Epoch: 1303, cost 14.30 s 2023-09-04 07:53:27,162 44k INFO ====> Epoch: 1304, cost 13.95 s 2023-09-04 07:53:38,106 44k INFO Train Epoch: 1305 [71%] 2023-09-04 07:53:38,106 44k INFO Losses: [2.5814831256866455, 2.542598247528076, 6.278233528137207, 15.13090991973877, 0.8271403908729553], step: 27400, lr: 8.495825325270724e-05, reference_loss: 27.360366821289062 2023-09-04 07:53:41,512 44k INFO ====> Epoch: 1305, cost 14.35 s 2023-09-04 07:53:56,050 44k INFO ====> Epoch: 1306, cost 14.54 s 2023-09-04 07:54:10,443 44k INFO ====> Epoch: 1307, cost 14.39 s 2023-09-04 07:54:24,627 44k INFO ====> Epoch: 1308, cost 14.18 s 2023-09-04 07:54:38,803 44k INFO ====> Epoch: 1309, cost 14.18 s 2023-09-04 07:54:53,060 44k INFO ====> Epoch: 1310, cost 14.26 s 2023-09-04 07:55:07,414 44k INFO ====> Epoch: 1311, cost 14.35 s 2023-09-04 07:55:21,514 44k INFO ====> Epoch: 1312, cost 14.10 s 2023-09-04 07:55:35,411 44k INFO ====> Epoch: 1313, cost 13.90 s 2023-09-04 07:55:49,590 44k INFO ====> Epoch: 1314, cost 14.18 s 2023-09-04 07:55:53,702 44k INFO Train Epoch: 1315 [24%] 2023-09-04 07:55:53,702 44k INFO Losses: [2.2332067489624023, 2.7679357528686523, 6.896419048309326, 16.530075073242188, 0.5758401155471802], step: 27600, lr: 8.48521151525054e-05, reference_loss: 29.003477096557617 2023-09-04 07:56:04,050 44k INFO ====> Epoch: 1315, cost 14.46 s 2023-09-04 07:56:18,295 44k INFO ====> Epoch: 1316, cost 14.24 s 2023-09-04 07:56:32,417 44k INFO ====> Epoch: 1317, cost 14.12 s 2023-09-04 07:56:46,575 44k INFO ====> Epoch: 1318, cost 14.16 s 2023-09-04 07:57:00,895 44k INFO ====> Epoch: 1319, cost 14.32 s 2023-09-04 07:57:14,983 44k INFO ====> Epoch: 1320, cost 14.09 s 2023-09-04 07:57:29,066 44k INFO ====> Epoch: 1321, cost 14.08 s 2023-09-04 07:57:43,046 44k INFO ====> Epoch: 1322, cost 13.98 s 2023-09-04 07:57:57,227 44k INFO ====> Epoch: 1323, cost 14.18 s 2023-09-04 07:58:08,922 44k INFO Train Epoch: 1324 [76%] 2023-09-04 07:58:08,923 44k INFO Losses: [2.1393260955810547, 2.610600233078003, 6.4770965576171875, 15.661272048950195, 0.5349127650260925], step: 27800, lr: 8.47567042383551e-05, reference_loss: 27.423206329345703 2023-09-04 07:58:11,656 44k INFO ====> Epoch: 1324, cost 14.43 s 2023-09-04 07:58:25,700 44k INFO ====> Epoch: 1325, cost 14.04 s 2023-09-04 07:58:39,727 44k INFO ====> Epoch: 1326, cost 14.03 s 2023-09-04 07:58:53,929 44k INFO ====> Epoch: 1327, cost 14.20 s 2023-09-04 07:59:08,149 44k INFO ====> Epoch: 1328, cost 14.22 s 2023-09-04 07:59:22,279 44k INFO ====> Epoch: 1329, cost 14.13 s 2023-09-04 07:59:36,323 44k INFO ====> Epoch: 1330, cost 14.04 s 2023-09-04 07:59:50,558 44k INFO ====> Epoch: 1331, cost 14.24 s 2023-09-04 08:00:04,762 44k INFO ====> Epoch: 1332, cost 14.20 s 2023-09-04 08:00:18,953 44k INFO ====> Epoch: 1333, cost 14.19 s 2023-09-04 08:00:23,649 44k INFO Train Epoch: 1334 [29%] 2023-09-04 08:00:23,649 44k INFO Losses: [2.4376204013824463, 2.3192105293273926, 7.519189834594727, 17.869291305541992, 0.7957101464271545], step: 28000, lr: 8.465081793275431e-05, reference_loss: 30.941022872924805 2023-09-04 08:00:28,304 44k INFO Saving model and optimizer state at iteration 1334 to ./logs\44k\G_28000.pth 2023-09-04 08:00:28,833 44k INFO Saving model and optimizer state at iteration 1334 to ./logs\44k\D_28000.pth 2023-09-04 08:00:30,148 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_25600.pth 2023-09-04 08:00:30,181 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_25600.pth 2023-09-04 08:00:39,545 44k INFO ====> Epoch: 1334, cost 20.59 s 2023-09-04 08:00:53,996 44k INFO ====> Epoch: 1335, cost 14.45 s 2023-09-04 08:01:08,247 44k INFO ====> Epoch: 1336, cost 14.25 s 2023-09-04 08:01:22,228 44k INFO ====> Epoch: 1337, cost 13.98 s 2023-09-04 08:01:36,192 44k INFO ====> Epoch: 1338, cost 13.96 s 2023-09-04 08:01:50,509 44k INFO ====> Epoch: 1339, cost 14.32 s 2023-09-04 08:02:04,921 44k INFO ====> Epoch: 1340, cost 14.41 s 2023-09-04 08:02:19,262 44k INFO ====> Epoch: 1341, cost 14.34 s 2023-09-04 08:02:33,298 44k INFO ====> Epoch: 1342, cost 14.04 s 2023-09-04 08:02:45,715 44k INFO Train Epoch: 1343 [81%] 2023-09-04 08:02:45,715 44k INFO Losses: [2.4673986434936523, 2.3633439540863037, 6.945917129516602, 17.23363494873047, 0.7074106335639954], step: 28200, lr: 8.455563336477959e-05, reference_loss: 29.71770668029785 2023-09-04 08:02:47,769 44k INFO ====> Epoch: 1343, cost 14.47 s 2023-09-04 08:03:02,008 44k INFO ====> Epoch: 1344, cost 14.24 s 2023-09-04 08:03:16,194 44k INFO ====> Epoch: 1345, cost 14.19 s 2023-09-04 08:03:30,300 44k INFO ====> Epoch: 1346, cost 14.11 s 2023-09-04 08:03:44,304 44k INFO ====> Epoch: 1347, cost 14.00 s 2023-09-04 08:03:58,859 44k INFO ====> Epoch: 1348, cost 14.55 s 2023-09-04 08:04:13,040 44k INFO ====> Epoch: 1349, cost 14.18 s 2023-09-04 08:04:27,050 44k INFO ====> Epoch: 1350, cost 14.01 s 2023-09-04 08:04:41,293 44k INFO ====> Epoch: 1351, cost 14.24 s 2023-09-04 08:04:55,599 44k INFO ====> Epoch: 1352, cost 14.31 s 2023-09-04 08:05:01,079 44k INFO Train Epoch: 1353 [33%] 2023-09-04 08:05:01,079 44k INFO Losses: [2.403240919113159, 2.54768705368042, 7.100437641143799, 17.355337142944336, 0.6975741386413574], step: 28400, lr: 8.444999825643989e-05, reference_loss: 30.104276657104492 2023-09-04 08:05:10,115 44k INFO ====> Epoch: 1353, cost 14.52 s 2023-09-04 08:05:24,168 44k INFO ====> Epoch: 1354, cost 14.05 s 2023-09-04 08:05:38,018 44k INFO ====> Epoch: 1355, cost 13.85 s 2023-09-04 08:05:52,382 44k INFO ====> Epoch: 1356, cost 14.36 s 2023-09-04 08:06:06,741 44k INFO ====> Epoch: 1357, cost 14.36 s 2023-09-04 08:06:20,953 44k INFO ====> Epoch: 1358, cost 14.21 s 2023-09-04 08:06:35,209 44k INFO ====> Epoch: 1359, cost 14.26 s 2023-09-04 08:06:49,345 44k INFO ====> Epoch: 1360, cost 14.14 s 2023-09-04 08:07:03,728 44k INFO ====> Epoch: 1361, cost 14.38 s 2023-09-04 08:07:16,638 44k INFO Train Epoch: 1362 [86%] 2023-09-04 08:07:16,638 44k INFO Losses: [2.2767767906188965, 2.8095455169677734, 7.87397575378418, 16.038301467895508, 0.7109388113021851], step: 28600, lr: 8.43550394976729e-05, reference_loss: 29.70953941345215 2023-09-04 08:07:17,937 44k INFO ====> Epoch: 1362, cost 14.21 s 2023-09-04 08:07:32,107 44k INFO ====> Epoch: 1363, cost 14.17 s 2023-09-04 08:07:46,270 44k INFO ====> Epoch: 1364, cost 14.16 s 2023-09-04 08:08:00,632 44k INFO ====> Epoch: 1365, cost 14.36 s 2023-09-04 08:08:14,860 44k INFO ====> Epoch: 1366, cost 14.23 s 2023-09-04 08:08:28,859 44k INFO ====> Epoch: 1367, cost 14.00 s 2023-09-04 08:08:42,983 44k INFO ====> Epoch: 1368, cost 14.12 s 2023-09-04 08:08:57,122 44k INFO ====> Epoch: 1369, cost 14.14 s 2023-09-04 08:09:11,452 44k INFO ====> Epoch: 1370, cost 14.33 s 2023-09-04 08:09:25,520 44k INFO ====> Epoch: 1371, cost 14.07 s 2023-09-04 08:09:31,632 44k INFO Train Epoch: 1372 [38%] 2023-09-04 08:09:31,633 44k INFO Losses: [2.348954200744629, 2.4550538063049316, 6.580178737640381, 14.99605655670166, 0.7040591239929199], step: 28800, lr: 8.424965499067151e-05, reference_loss: 27.08430290222168 2023-09-04 08:09:36,452 44k INFO Saving model and optimizer state at iteration 1372 to ./logs\44k\G_28800.pth 2023-09-04 08:09:36,971 44k INFO Saving model and optimizer state at iteration 1372 to ./logs\44k\D_28800.pth 2023-09-04 08:09:38,639 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_26400.pth 2023-09-04 08:09:38,670 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_26400.pth 2023-09-04 08:09:46,732 44k INFO ====> Epoch: 1372, cost 21.21 s 2023-09-04 08:10:00,980 44k INFO ====> Epoch: 1373, cost 14.25 s 2023-09-04 08:10:14,980 44k INFO ====> Epoch: 1374, cost 14.00 s 2023-09-04 08:10:29,062 44k INFO ====> Epoch: 1375, cost 14.08 s 2023-09-04 08:10:43,116 44k INFO ====> Epoch: 1376, cost 14.05 s 2023-09-04 08:10:57,360 44k INFO ====> Epoch: 1377, cost 14.24 s 2023-09-04 08:11:11,626 44k INFO ====> Epoch: 1378, cost 14.27 s 2023-09-04 08:11:25,580 44k INFO ====> Epoch: 1379, cost 13.95 s 2023-09-04 08:11:39,564 44k INFO ====> Epoch: 1380, cost 13.98 s 2023-09-04 08:11:53,506 44k INFO Train Epoch: 1381 [90%] 2023-09-04 08:11:53,507 44k INFO Losses: [2.4761152267456055, 2.6169886589050293, 7.178109169006348, 17.701284408569336, 0.8752359747886658], step: 29000, lr: 8.415492150541829e-05, reference_loss: 30.847734451293945 2023-09-04 08:11:54,263 44k INFO ====> Epoch: 1381, cost 14.70 s 2023-09-04 08:12:08,662 44k INFO ====> Epoch: 1382, cost 14.40 s 2023-09-04 08:12:22,819 44k INFO ====> Epoch: 1383, cost 14.16 s 2023-09-04 08:12:37,036 44k INFO ====> Epoch: 1384, cost 14.22 s 2023-09-04 08:12:51,419 44k INFO ====> Epoch: 1385, cost 14.38 s 2023-09-04 08:13:05,760 44k INFO ====> Epoch: 1386, cost 14.34 s 2023-09-04 08:13:19,938 44k INFO ====> Epoch: 1387, cost 14.18 s 2023-09-04 08:13:33,984 44k INFO ====> Epoch: 1388, cost 14.05 s 2023-09-04 08:13:48,050 44k INFO ====> Epoch: 1389, cost 14.07 s 2023-09-04 08:14:02,304 44k INFO ====> Epoch: 1390, cost 14.25 s 2023-09-04 08:14:09,184 44k INFO Train Epoch: 1391 [43%] 2023-09-04 08:14:09,184 44k INFO Losses: [2.396435260772705, 2.4615559577941895, 5.555205345153809, 15.235918045043945, 0.6457263827323914], step: 29200, lr: 8.404978700524619e-05, reference_loss: 26.294841766357422 2023-09-04 08:14:16,792 44k INFO ====> Epoch: 1391, cost 14.49 s 2023-09-04 08:14:30,865 44k INFO ====> Epoch: 1392, cost 14.07 s 2023-09-04 08:14:45,006 44k INFO ====> Epoch: 1393, cost 14.14 s 2023-09-04 08:14:59,176 44k INFO ====> Epoch: 1394, cost 14.17 s 2023-09-04 08:15:13,376 44k INFO ====> Epoch: 1395, cost 14.20 s 2023-09-04 08:15:27,308 44k INFO ====> Epoch: 1396, cost 13.93 s 2023-09-04 08:15:41,320 44k INFO ====> Epoch: 1397, cost 14.01 s 2023-09-04 08:15:55,516 44k INFO ====> Epoch: 1398, cost 14.20 s 2023-09-04 08:16:09,593 44k INFO ====> Epoch: 1399, cost 14.08 s 2023-09-04 08:16:23,535 44k INFO Train Epoch: 1400 [95%] 2023-09-04 08:16:23,536 44k INFO Losses: [2.3571841716766357, 2.4519431591033936, 6.862038612365723, 15.003006935119629, 0.5602516531944275], step: 29400, lr: 8.395527825908361e-05, reference_loss: 27.23442268371582 2023-09-04 08:16:23,786 44k INFO ====> Epoch: 1400, cost 14.19 s 2023-09-04 08:16:37,754 44k INFO ====> Epoch: 1401, cost 13.97 s 2023-09-04 08:16:51,956 44k INFO ====> Epoch: 1402, cost 14.20 s 2023-09-04 08:17:06,140 44k INFO ====> Epoch: 1403, cost 14.18 s 2023-09-04 08:17:20,280 44k INFO ====> Epoch: 1404, cost 14.14 s 2023-09-04 08:17:34,445 44k INFO ====> Epoch: 1405, cost 14.16 s 2023-09-04 08:17:48,625 44k INFO ====> Epoch: 1406, cost 14.18 s 2023-09-04 08:18:03,087 44k INFO ====> Epoch: 1407, cost 14.46 s 2023-09-04 08:18:17,477 44k INFO ====> Epoch: 1408, cost 14.39 s 2023-09-04 08:18:31,619 44k INFO ====> Epoch: 1409, cost 14.14 s 2023-09-04 08:18:39,100 44k INFO Train Epoch: 1410 [48%] 2023-09-04 08:18:39,100 44k INFO Losses: [2.398773193359375, 2.682448387145996, 8.212663650512695, 18.63374137878418, 0.7600454688072205], step: 29600, lr: 8.385039317264206e-05, reference_loss: 32.68767166137695 2023-09-04 08:18:43,756 44k INFO Saving model and optimizer state at iteration 1410 to ./logs\44k\G_29600.pth 2023-09-04 08:18:44,402 44k INFO Saving model and optimizer state at iteration 1410 to ./logs\44k\D_29600.pth 2023-09-04 08:18:45,560 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_27200.pth 2023-09-04 08:18:45,592 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_27200.pth 2023-09-04 08:18:52,225 44k INFO ====> Epoch: 1410, cost 20.61 s 2023-09-04 08:19:06,521 44k INFO ====> Epoch: 1411, cost 14.30 s 2023-09-04 08:19:20,646 44k INFO ====> Epoch: 1412, cost 14.13 s 2023-09-04 08:19:34,476 44k INFO ====> Epoch: 1413, cost 13.83 s 2023-09-04 08:19:48,487 44k INFO ====> Epoch: 1414, cost 14.01 s 2023-09-04 08:20:02,624 44k INFO ====> Epoch: 1415, cost 14.14 s 2023-09-04 08:20:16,959 44k INFO ====> Epoch: 1416, cost 14.34 s 2023-09-04 08:20:30,985 44k INFO ====> Epoch: 1417, cost 14.03 s 2023-09-04 08:20:45,107 44k INFO ====> Epoch: 1418, cost 14.12 s 2023-09-04 08:20:59,198 44k INFO ====> Epoch: 1419, cost 14.09 s 2023-09-04 08:20:59,907 44k INFO Train Epoch: 1420 [0%] 2023-09-04 08:20:59,907 44k INFO Losses: [2.454202651977539, 2.3779642581939697, 5.983738422393799, 15.743879318237305, 0.6502825617790222], step: 29800, lr: 8.374563911883578e-05, reference_loss: 27.210065841674805 2023-09-04 08:21:13,520 44k INFO ====> Epoch: 1420, cost 14.32 s 2023-09-04 08:21:27,445 44k INFO ====> Epoch: 1421, cost 13.92 s 2023-09-04 08:21:41,585 44k INFO ====> Epoch: 1422, cost 14.14 s 2023-09-04 08:21:55,815 44k INFO ====> Epoch: 1423, cost 14.23 s 2023-09-04 08:22:09,902 44k INFO ====> Epoch: 1424, cost 14.09 s 2023-09-04 08:22:23,838 44k INFO ====> Epoch: 1425, cost 13.94 s 2023-09-04 08:22:37,927 44k INFO ====> Epoch: 1426, cost 14.09 s 2023-09-04 08:22:52,203 44k INFO ====> Epoch: 1427, cost 14.27 s 2023-09-04 08:23:06,369 44k INFO ====> Epoch: 1428, cost 14.17 s 2023-09-04 08:23:14,513 44k INFO Train Epoch: 1429 [52%] 2023-09-04 08:23:14,513 44k INFO Losses: [2.534501552581787, 2.782521963119507, 6.28605318069458, 16.6182918548584, 0.8017011880874634], step: 30000, lr: 8.365147236801214e-05, reference_loss: 29.023069381713867 2023-09-04 08:23:20,558 44k INFO ====> Epoch: 1429, cost 14.19 s 2023-09-04 08:23:34,440 44k INFO ====> Epoch: 1430, cost 13.88 s 2023-09-04 08:23:48,706 44k INFO ====> Epoch: 1431, cost 14.27 s 2023-09-04 08:24:03,042 44k INFO ====> Epoch: 1432, cost 14.34 s 2023-09-04 08:24:17,318 44k INFO ====> Epoch: 1433, cost 14.28 s 2023-09-04 08:24:31,569 44k INFO ====> Epoch: 1434, cost 14.25 s 2023-09-04 08:24:45,726 44k INFO ====> Epoch: 1435, cost 14.16 s 2023-09-04 08:24:59,995 44k INFO ====> Epoch: 1436, cost 14.27 s 2023-09-04 08:25:14,216 44k INFO ====> Epoch: 1437, cost 14.22 s 2023-09-04 08:25:28,111 44k INFO ====> Epoch: 1438, cost 13.90 s 2023-09-04 08:25:29,488 44k INFO Train Epoch: 1439 [5%] 2023-09-04 08:25:29,489 44k INFO Losses: [2.305063009262085, 2.280881404876709, 6.675364971160889, 17.48219108581543, 0.6845791339874268], step: 30200, lr: 8.354696682539207e-05, reference_loss: 29.42807960510254 2023-09-04 08:25:42,441 44k INFO ====> Epoch: 1439, cost 14.33 s 2023-09-04 08:25:56,819 44k INFO ====> Epoch: 1440, cost 14.38 s 2023-09-04 08:26:11,066 44k INFO ====> Epoch: 1441, cost 14.25 s 2023-09-04 08:26:25,127 44k INFO ====> Epoch: 1442, cost 14.06 s 2023-09-04 08:26:39,291 44k INFO ====> Epoch: 1443, cost 14.16 s 2023-09-04 08:26:53,534 44k INFO ====> Epoch: 1444, cost 14.24 s 2023-09-04 08:27:07,721 44k INFO ====> Epoch: 1445, cost 14.19 s 2023-09-04 08:27:21,813 44k INFO ====> Epoch: 1446, cost 14.09 s 2023-09-04 08:27:35,664 44k INFO ====> Epoch: 1447, cost 13.85 s 2023-09-04 08:27:44,545 44k INFO Train Epoch: 1448 [57%] 2023-09-04 08:27:44,546 44k INFO Losses: [2.2949771881103516, 2.5410873889923096, 7.775840759277344, 15.970759391784668, 0.8848915696144104], step: 30400, lr: 8.345302346917795e-05, reference_loss: 29.46755599975586 2023-09-04 08:27:49,315 44k INFO Saving model and optimizer state at iteration 1448 to ./logs\44k\G_30400.pth 2023-09-04 08:27:49,964 44k INFO Saving model and optimizer state at iteration 1448 to ./logs\44k\D_30400.pth 2023-09-04 08:27:51,092 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_28000.pth 2023-09-04 08:27:51,125 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_28000.pth 2023-09-04 08:27:56,391 44k INFO ====> Epoch: 1448, cost 20.73 s 2023-09-04 08:28:10,582 44k INFO ====> Epoch: 1449, cost 14.19 s 2023-09-04 08:28:24,683 44k INFO ====> Epoch: 1450, cost 14.10 s 2023-09-04 08:28:38,647 44k INFO ====> Epoch: 1451, cost 13.96 s 2023-09-04 08:28:52,894 44k INFO ====> Epoch: 1452, cost 14.25 s 2023-09-04 08:29:07,059 44k INFO ====> Epoch: 1453, cost 14.16 s 2023-09-04 08:29:20,962 44k INFO ====> Epoch: 1454, cost 13.90 s 2023-09-04 08:29:34,855 44k INFO ====> Epoch: 1455, cost 13.89 s 2023-09-04 08:29:48,987 44k INFO ====> Epoch: 1456, cost 14.13 s 2023-09-04 08:30:03,408 44k INFO ====> Epoch: 1457, cost 14.42 s 2023-09-04 08:30:05,534 44k INFO Train Epoch: 1458 [10%] 2023-09-04 08:30:05,535 44k INFO Losses: [2.539870500564575, 2.275099754333496, 6.300292015075684, 15.438322067260742, 0.7234359383583069], step: 30600, lr: 8.334876584819357e-05, reference_loss: 27.277021408081055 2023-09-04 08:30:17,962 44k INFO ====> Epoch: 1458, cost 14.55 s 2023-09-04 08:30:32,060 44k INFO ====> Epoch: 1459, cost 14.10 s 2023-09-04 08:30:46,081 44k INFO ====> Epoch: 1460, cost 14.02 s 2023-09-04 08:31:00,395 44k INFO ====> Epoch: 1461, cost 14.31 s 2023-09-04 08:31:14,618 44k INFO ====> Epoch: 1462, cost 14.22 s 2023-09-04 08:31:28,606 44k INFO ====> Epoch: 1463, cost 13.99 s 2023-09-04 08:31:42,674 44k INFO ====> Epoch: 1464, cost 14.07 s 2023-09-04 08:31:56,990 44k INFO ====> Epoch: 1465, cost 14.32 s 2023-09-04 08:32:11,260 44k INFO ====> Epoch: 1466, cost 14.27 s 2023-09-04 08:32:20,847 44k INFO Train Epoch: 1467 [62%] 2023-09-04 08:32:20,847 44k INFO Losses: [2.4846479892730713, 2.456509590148926, 7.119597911834717, 17.785350799560547, 0.6759963631629944], step: 30800, lr: 8.325504535662326e-05, reference_loss: 30.52210235595703 2023-09-04 08:32:25,582 44k INFO ====> Epoch: 1467, cost 14.32 s 2023-09-04 08:32:39,772 44k INFO ====> Epoch: 1468, cost 14.19 s 2023-09-04 08:32:53,811 44k INFO ====> Epoch: 1469, cost 14.04 s 2023-09-04 08:33:07,989 44k INFO ====> Epoch: 1470, cost 14.18 s 2023-09-04 08:33:22,085 44k INFO ====> Epoch: 1471, cost 14.10 s 2023-09-04 08:33:36,088 44k INFO ====> Epoch: 1472, cost 14.00 s 2023-09-04 08:33:50,209 44k INFO ====> Epoch: 1473, cost 14.12 s 2023-09-04 08:34:04,285 44k INFO ====> Epoch: 1474, cost 14.08 s 2023-09-04 08:34:18,431 44k INFO ====> Epoch: 1475, cost 14.15 s 2023-09-04 08:34:32,381 44k INFO ====> Epoch: 1476, cost 13.95 s 2023-09-04 08:34:35,056 44k INFO Train Epoch: 1477 [14%] 2023-09-04 08:34:35,056 44k INFO Losses: [2.301278829574585, 2.4505743980407715, 7.449445724487305, 16.723243713378906, 0.7062522172927856], step: 31000, lr: 8.315103506912256e-05, reference_loss: 29.630794525146484 2023-09-04 08:34:46,876 44k INFO ====> Epoch: 1477, cost 14.49 s 2023-09-04 08:35:01,199 44k INFO ====> Epoch: 1478, cost 14.32 s 2023-09-04 08:35:15,349 44k INFO ====> Epoch: 1479, cost 14.15 s 2023-09-04 08:35:29,342 44k INFO ====> Epoch: 1480, cost 13.99 s 2023-09-04 08:35:43,522 44k INFO ====> Epoch: 1481, cost 14.18 s 2023-09-04 08:35:57,864 44k INFO ====> Epoch: 1482, cost 14.34 s 2023-09-04 08:36:12,331 44k INFO ====> Epoch: 1483, cost 14.47 s 2023-09-04 08:36:26,629 44k INFO ====> Epoch: 1484, cost 14.30 s 2023-09-04 08:36:40,700 44k INFO ====> Epoch: 1485, cost 14.07 s 2023-09-04 08:36:51,192 44k INFO Train Epoch: 1486 [67%] 2023-09-04 08:36:51,192 44k INFO Losses: [2.2368946075439453, 2.673727035522461, 7.968774318695068, 18.591529846191406, 0.6583554148674011], step: 31200, lr: 8.305753691348759e-05, reference_loss: 32.12928009033203 2023-09-04 08:36:55,870 44k INFO Saving model and optimizer state at iteration 1486 to ./logs\44k\G_31200.pth 2023-09-04 08:36:56,506 44k INFO Saving model and optimizer state at iteration 1486 to ./logs\44k\D_31200.pth 2023-09-04 08:36:57,632 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_28800.pth 2023-09-04 08:36:57,664 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_28800.pth 2023-09-04 08:37:01,619 44k INFO ====> Epoch: 1486, cost 20.92 s 2023-09-04 08:37:15,803 44k INFO ====> Epoch: 1487, cost 14.18 s 2023-09-04 08:37:29,681 44k INFO ====> Epoch: 1488, cost 13.88 s 2023-09-04 08:37:43,678 44k INFO ====> Epoch: 1489, cost 14.00 s 2023-09-04 08:37:57,955 44k INFO ====> Epoch: 1490, cost 14.28 s 2023-09-04 08:38:12,201 44k INFO ====> Epoch: 1491, cost 14.25 s 2023-09-04 08:38:26,278 44k INFO ====> Epoch: 1492, cost 14.08 s 2023-09-04 08:38:40,333 44k INFO ====> Epoch: 1493, cost 14.05 s 2023-09-04 08:38:54,602 44k INFO ====> Epoch: 1494, cost 14.27 s 2023-09-04 08:39:08,642 44k INFO ====> Epoch: 1495, cost 14.04 s 2023-09-04 08:39:12,038 44k INFO Train Epoch: 1496 [19%] 2023-09-04 08:39:12,039 44k INFO Losses: [2.4450478553771973, 2.478935718536377, 5.981303691864014, 16.813566207885742, 0.7584121227264404], step: 31400, lr: 8.295377337271398e-05, reference_loss: 28.477264404296875 2023-09-04 08:39:22,800 44k INFO ====> Epoch: 1496, cost 14.16 s 2023-09-04 08:39:36,865 44k INFO ====> Epoch: 1497, cost 14.07 s 2023-09-04 08:39:50,908 44k INFO ====> Epoch: 1498, cost 14.04 s 2023-09-04 08:40:05,077 44k INFO ====> Epoch: 1499, cost 14.17 s 2023-09-04 08:40:19,132 44k INFO ====> Epoch: 1500, cost 14.05 s 2023-09-04 08:40:33,204 44k INFO ====> Epoch: 1501, cost 14.07 s 2023-09-04 08:40:47,281 44k INFO ====> Epoch: 1502, cost 14.08 s 2023-09-04 08:41:01,653 44k INFO ====> Epoch: 1503, cost 14.37 s 2023-09-04 08:41:15,911 44k INFO ====> Epoch: 1504, cost 14.26 s 2023-09-04 08:41:26,834 44k INFO Train Epoch: 1505 [71%] 2023-09-04 08:41:26,834 44k INFO Losses: [2.543879985809326, 2.1307058334350586, 6.225126266479492, 15.349884986877441, 0.6594100594520569], step: 31600, lr: 8.28604970255601e-05, reference_loss: 26.909006118774414 2023-09-04 08:41:30,236 44k INFO ====> Epoch: 1505, cost 14.32 s 2023-09-04 08:41:44,411 44k INFO ====> Epoch: 1506, cost 14.18 s 2023-09-04 08:41:58,783 44k INFO ====> Epoch: 1507, cost 14.37 s 2023-09-04 08:42:13,100 44k INFO ====> Epoch: 1508, cost 14.32 s 2023-09-04 08:42:27,224 44k INFO ====> Epoch: 1509, cost 14.12 s 2023-09-04 08:42:41,243 44k INFO ====> Epoch: 1510, cost 14.02 s 2023-09-04 08:42:55,487 44k INFO ====> Epoch: 1511, cost 14.24 s 2023-09-04 08:43:09,629 44k INFO ====> Epoch: 1512, cost 14.14 s 2023-09-04 08:43:23,659 44k INFO ====> Epoch: 1513, cost 14.03 s 2023-09-04 08:43:37,711 44k INFO ====> Epoch: 1514, cost 14.05 s 2023-09-04 08:43:41,804 44k INFO Train Epoch: 1515 [24%] 2023-09-04 08:43:41,804 44k INFO Losses: [2.527080774307251, 2.3096790313720703, 5.941561698913574, 15.383397102355957, 0.6376937031745911], step: 31800, lr: 8.275697964614889e-05, reference_loss: 26.79941177368164 2023-09-04 08:43:52,131 44k INFO ====> Epoch: 1515, cost 14.42 s 2023-09-04 08:44:06,496 44k INFO ====> Epoch: 1516, cost 14.37 s 2023-09-04 08:44:20,684 44k INFO ====> Epoch: 1517, cost 14.19 s 2023-09-04 08:44:34,499 44k INFO ====> Epoch: 1518, cost 13.81 s 2023-09-04 08:44:48,659 44k INFO ====> Epoch: 1519, cost 14.16 s 2023-09-04 08:45:02,941 44k INFO ====> Epoch: 1520, cost 14.28 s 2023-09-04 08:45:17,017 44k INFO ====> Epoch: 1521, cost 14.08 s 2023-09-04 08:45:31,035 44k INFO ====> Epoch: 1522, cost 14.02 s 2023-09-04 08:45:45,193 44k INFO ====> Epoch: 1523, cost 14.16 s 2023-09-04 08:45:56,984 44k INFO Train Epoch: 1524 [76%] 2023-09-04 08:45:56,984 44k INFO Losses: [2.3143045902252197, 2.446739673614502, 10.072339057922363, 18.299421310424805, 0.7271527647972107], step: 32000, lr: 8.266392458127321e-05, reference_loss: 33.85995864868164 2023-09-04 08:46:01,689 44k INFO Saving model and optimizer state at iteration 1524 to ./logs\44k\G_32000.pth 2023-09-04 08:46:02,254 44k INFO Saving model and optimizer state at iteration 1524 to ./logs\44k\D_32000.pth 2023-09-04 08:46:03,431 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_29600.pth 2023-09-04 08:46:03,466 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_29600.pth 2023-09-04 08:46:06,031 44k INFO ====> Epoch: 1524, cost 20.84 s 2023-09-04 08:46:20,079 44k INFO ====> Epoch: 1525, cost 14.05 s 2023-09-04 08:46:34,180 44k INFO ====> Epoch: 1526, cost 14.10 s 2023-09-04 08:46:48,202 44k INFO ====> Epoch: 1527, cost 14.02 s 2023-09-04 08:47:02,390 44k INFO ====> Epoch: 1528, cost 14.19 s 2023-09-04 08:47:16,431 44k INFO ====> Epoch: 1529, cost 14.04 s 2023-09-04 08:47:30,490 44k INFO ====> Epoch: 1530, cost 14.06 s 2023-09-04 08:47:44,488 44k INFO ====> Epoch: 1531, cost 14.00 s 2023-09-04 08:47:58,890 44k INFO ====> Epoch: 1532, cost 14.40 s 2023-09-04 08:48:13,299 44k INFO ====> Epoch: 1533, cost 14.41 s 2023-09-04 08:48:18,023 44k INFO Train Epoch: 1534 [29%] 2023-09-04 08:48:18,023 44k INFO Losses: [2.482541084289551, 2.2023086547851562, 5.9423112869262695, 15.740888595581055, 0.6525241136550903], step: 32200, lr: 8.256065277924843e-05, reference_loss: 27.02057456970215 2023-09-04 08:48:27,550 44k INFO ====> Epoch: 1534, cost 14.25 s 2023-09-04 08:48:41,567 44k INFO ====> Epoch: 1535, cost 14.02 s 2023-09-04 08:48:55,859 44k INFO ====> Epoch: 1536, cost 14.29 s 2023-09-04 08:49:10,325 44k INFO ====> Epoch: 1537, cost 14.47 s 2023-09-04 08:49:24,456 44k INFO ====> Epoch: 1538, cost 14.13 s 2023-09-04 08:49:38,408 44k INFO ====> Epoch: 1539, cost 13.95 s 2023-09-04 08:49:52,606 44k INFO ====> Epoch: 1540, cost 14.20 s 2023-09-04 08:50:06,961 44k INFO ====> Epoch: 1541, cost 14.35 s 2023-09-04 08:50:20,935 44k INFO ====> Epoch: 1542, cost 13.97 s 2023-09-04 08:50:33,249 44k INFO Train Epoch: 1543 [81%] 2023-09-04 08:50:33,249 44k INFO Losses: [2.589852809906006, 2.188912868499756, 6.771442413330078, 16.105863571166992, 0.672415554523468], step: 32400, lr: 8.246781847169636e-05, reference_loss: 28.3284854888916 2023-09-04 08:50:35,252 44k INFO ====> Epoch: 1543, cost 14.32 s 2023-09-04 08:50:49,586 44k INFO ====> Epoch: 1544, cost 14.33 s 2023-09-04 08:51:03,743 44k INFO ====> Epoch: 1545, cost 14.16 s 2023-09-04 08:51:17,859 44k INFO ====> Epoch: 1546, cost 14.12 s 2023-09-04 08:51:31,932 44k INFO ====> Epoch: 1547, cost 14.07 s 2023-09-04 08:51:46,112 44k INFO ====> Epoch: 1548, cost 14.18 s 2023-09-04 08:52:00,331 44k INFO ====> Epoch: 1549, cost 14.22 s 2023-09-04 08:52:14,485 44k INFO ====> Epoch: 1550, cost 14.15 s 2023-09-04 08:52:28,525 44k INFO ====> Epoch: 1551, cost 14.04 s 2023-09-04 08:52:42,689 44k INFO ====> Epoch: 1552, cost 14.16 s 2023-09-04 08:52:48,238 44k INFO Train Epoch: 1553 [33%] 2023-09-04 08:52:48,238 44k INFO Losses: [2.429267406463623, 2.2384543418884277, 5.999950885772705, 15.591471672058105, 0.6608731746673584], step: 32600, lr: 8.236479166446738e-05, reference_loss: 26.92001724243164 2023-09-04 08:52:57,227 44k INFO ====> Epoch: 1553, cost 14.54 s 2023-09-04 08:53:11,629 44k INFO ====> Epoch: 1554, cost 14.40 s 2023-09-04 08:53:25,497 44k INFO ====> Epoch: 1555, cost 13.87 s 2023-09-04 08:53:39,737 44k INFO ====> Epoch: 1556, cost 14.24 s 2023-09-04 08:53:54,262 44k INFO ====> Epoch: 1557, cost 14.53 s 2023-09-04 08:54:08,780 44k INFO ====> Epoch: 1558, cost 14.52 s 2023-09-04 08:54:23,007 44k INFO ====> Epoch: 1559, cost 14.23 s 2023-09-04 08:54:37,067 44k INFO ====> Epoch: 1560, cost 14.06 s 2023-09-04 08:54:51,444 44k INFO ====> Epoch: 1561, cost 14.38 s 2023-09-04 08:55:04,653 44k INFO Train Epoch: 1562 [86%] 2023-09-04 08:55:04,654 44k INFO Losses: [2.313096046447754, 2.6052541732788086, 8.487029075622559, 16.870769500732422, 0.6759847402572632], step: 32800, lr: 8.227217759052969e-05, reference_loss: 30.95213508605957 2023-09-04 08:55:09,336 44k INFO Saving model and optimizer state at iteration 1562 to ./logs\44k\G_32800.pth 2023-09-04 08:55:10,016 44k INFO Saving model and optimizer state at iteration 1562 to ./logs\44k\D_32800.pth 2023-09-04 08:55:11,077 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_30400.pth 2023-09-04 08:55:11,114 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_30400.pth 2023-09-04 08:55:12,251 44k INFO ====> Epoch: 1562, cost 20.81 s 2023-09-04 08:55:26,303 44k INFO ====> Epoch: 1563, cost 14.05 s 2023-09-04 08:55:40,434 44k INFO ====> Epoch: 1564, cost 14.13 s 2023-09-04 08:55:54,847 44k INFO ====> Epoch: 1565, cost 14.41 s 2023-09-04 08:56:09,083 44k INFO ====> Epoch: 1566, cost 14.24 s 2023-09-04 08:56:23,207 44k INFO ====> Epoch: 1567, cost 14.12 s 2023-09-04 08:56:37,310 44k INFO ====> Epoch: 1568, cost 14.10 s 2023-09-04 08:56:51,526 44k INFO ====> Epoch: 1569, cost 14.22 s 2023-09-04 08:57:05,652 44k INFO ====> Epoch: 1570, cost 14.13 s 2023-09-04 08:57:19,644 44k INFO ====> Epoch: 1571, cost 13.99 s 2023-09-04 08:57:25,853 44k INFO Train Epoch: 1572 [38%] 2023-09-04 08:57:25,853 44k INFO Losses: [2.306602954864502, 2.499236822128296, 7.132660388946533, 16.961288452148438, 0.6973729729652405], step: 33000, lr: 8.216939519688802e-05, reference_loss: 29.59716033935547 2023-09-04 08:57:33,968 44k INFO ====> Epoch: 1572, cost 14.32 s 2023-09-04 08:57:48,328 44k INFO ====> Epoch: 1573, cost 14.36 s 2023-09-04 08:58:02,471 44k INFO ====> Epoch: 1574, cost 14.14 s 2023-09-04 08:58:16,433 44k INFO ====> Epoch: 1575, cost 13.96 s 2023-09-04 08:58:30,546 44k INFO ====> Epoch: 1576, cost 14.11 s 2023-09-04 08:58:44,828 44k INFO ====> Epoch: 1577, cost 14.28 s 2023-09-04 08:58:58,847 44k INFO ====> Epoch: 1578, cost 14.02 s 2023-09-04 08:59:13,170 44k INFO ====> Epoch: 1579, cost 14.32 s 2023-09-04 08:59:27,237 44k INFO ====> Epoch: 1580, cost 14.07 s 2023-09-04 08:59:40,817 44k INFO Train Epoch: 1581 [90%] 2023-09-04 08:59:40,817 44k INFO Losses: [2.198019504547119, 2.8144891262054443, 7.151234149932861, 15.99543571472168, 0.5784316658973694], step: 33200, lr: 8.20770008340979e-05, reference_loss: 28.73760986328125 2023-09-04 08:59:41,473 44k INFO ====> Epoch: 1581, cost 14.24 s 2023-09-04 08:59:55,949 44k INFO ====> Epoch: 1582, cost 14.48 s 2023-09-04 09:00:10,373 44k INFO ====> Epoch: 1583, cost 14.42 s 2023-09-04 09:00:24,519 44k INFO ====> Epoch: 1584, cost 14.15 s 2023-09-04 09:00:38,635 44k INFO ====> Epoch: 1585, cost 14.12 s 2023-09-04 09:00:52,833 44k INFO ====> Epoch: 1586, cost 14.20 s 2023-09-04 09:01:07,147 44k INFO ====> Epoch: 1587, cost 14.31 s 2023-09-04 09:01:21,204 44k INFO ====> Epoch: 1588, cost 14.06 s 2023-09-04 09:01:35,260 44k INFO ====> Epoch: 1589, cost 14.06 s 2023-09-04 09:01:49,392 44k INFO ====> Epoch: 1590, cost 14.13 s 2023-09-04 09:01:56,242 44k INFO Train Epoch: 1591 [43%] 2023-09-04 09:01:56,242 44k INFO Losses: [2.3827786445617676, 2.394914150238037, 7.293755531311035, 17.516178131103516, 0.5956751108169556], step: 33400, lr: 8.197446227421386e-05, reference_loss: 30.18330192565918 2023-09-04 09:02:03,771 44k INFO ====> Epoch: 1591, cost 14.38 s 2023-09-04 09:02:18,008 44k INFO ====> Epoch: 1592, cost 14.24 s 2023-09-04 09:02:32,106 44k INFO ====> Epoch: 1593, cost 14.10 s 2023-09-04 09:02:46,268 44k INFO ====> Epoch: 1594, cost 14.16 s 2023-09-04 09:03:00,304 44k INFO ====> Epoch: 1595, cost 14.04 s 2023-09-04 09:03:14,405 44k INFO ====> Epoch: 1596, cost 14.10 s 2023-09-04 09:03:28,172 44k INFO ====> Epoch: 1597, cost 13.77 s 2023-09-04 09:03:42,400 44k INFO ====> Epoch: 1598, cost 14.23 s 2023-09-04 09:03:56,440 44k INFO ====> Epoch: 1599, cost 14.04 s 2023-09-04 09:04:10,923 44k INFO Train Epoch: 1600 [95%] 2023-09-04 09:04:10,923 44k INFO Losses: [2.2800800800323486, 2.5411553382873535, 7.666043281555176, 17.95314598083496, 0.36611077189445496], step: 33600, lr: 8.188228710134397e-05, reference_loss: 30.806535720825195 2023-09-04 09:04:15,626 44k INFO Saving model and optimizer state at iteration 1600 to ./logs\44k\G_33600.pth 2023-09-04 09:04:16,259 44k INFO Saving model and optimizer state at iteration 1600 to ./logs\44k\D_33600.pth 2023-09-04 09:04:17,396 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_31200.pth 2023-09-04 09:04:17,432 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_31200.pth 2023-09-04 09:04:17,432 44k INFO ====> Epoch: 1600, cost 20.99 s 2023-09-04 09:04:31,462 44k INFO ====> Epoch: 1601, cost 14.03 s 2023-09-04 09:04:45,564 44k INFO ====> Epoch: 1602, cost 14.10 s 2023-09-04 09:04:59,757 44k INFO ====> Epoch: 1603, cost 14.19 s 2023-09-04 09:05:13,950 44k INFO ====> Epoch: 1604, cost 14.19 s 2023-09-04 09:05:27,934 44k INFO ====> Epoch: 1605, cost 13.98 s 2023-09-04 09:05:41,862 44k INFO ====> Epoch: 1606, cost 13.93 s 2023-09-04 09:05:56,206 44k INFO ====> Epoch: 1607, cost 14.34 s 2023-09-04 09:06:10,536 44k INFO ====> Epoch: 1608, cost 14.33 s 2023-09-04 09:06:24,538 44k INFO ====> Epoch: 1609, cost 14.00 s 2023-09-04 09:06:32,062 44k INFO Train Epoch: 1610 [48%] 2023-09-04 09:06:32,062 44k INFO Losses: [2.176222324371338, 2.7920029163360596, 8.528916358947754, 17.748592376708984, 0.8019925951957703], step: 33800, lr: 8.177999179676341e-05, reference_loss: 32.0477294921875 2023-09-04 09:06:38,962 44k INFO ====> Epoch: 1610, cost 14.42 s 2023-09-04 09:06:53,320 44k INFO ====> Epoch: 1611, cost 14.36 s 2023-09-04 09:07:07,589 44k INFO ====> Epoch: 1612, cost 14.27 s 2023-09-04 09:07:21,565 44k INFO ====> Epoch: 1613, cost 13.98 s 2023-09-04 09:07:35,483 44k INFO ====> Epoch: 1614, cost 13.92 s 2023-09-04 09:07:49,639 44k INFO ====> Epoch: 1615, cost 14.16 s 2023-09-04 09:08:03,813 44k INFO ====> Epoch: 1616, cost 14.17 s 2023-09-04 09:08:18,049 44k INFO ====> Epoch: 1617, cost 14.24 s 2023-09-04 09:08:32,156 44k INFO ====> Epoch: 1618, cost 14.11 s 2023-09-04 09:08:46,343 44k INFO ====> Epoch: 1619, cost 14.19 s 2023-09-04 09:08:47,035 44k INFO Train Epoch: 1620 [0%] 2023-09-04 09:08:47,035 44k INFO Losses: [2.4085159301757812, 2.510188341140747, 6.153406620025635, 14.566841125488281, 0.6535301208496094], step: 34000, lr: 8.167782428941117e-05, reference_loss: 26.292482376098633 2023-09-04 09:09:00,722 44k INFO ====> Epoch: 1620, cost 14.38 s 2023-09-04 09:09:14,712 44k INFO ====> Epoch: 1621, cost 13.99 s 2023-09-04 09:09:28,722 44k INFO ====> Epoch: 1622, cost 14.01 s 2023-09-04 09:09:42,643 44k INFO ====> Epoch: 1623, cost 13.92 s 2023-09-04 09:09:56,677 44k INFO ====> Epoch: 1624, cost 14.03 s 2023-09-04 09:10:10,861 44k INFO ====> Epoch: 1625, cost 14.18 s 2023-09-04 09:10:24,898 44k INFO ====> Epoch: 1626, cost 14.04 s 2023-09-04 09:10:38,936 44k INFO ====> Epoch: 1627, cost 14.04 s 2023-09-04 09:10:53,109 44k INFO ====> Epoch: 1628, cost 14.17 s 2023-09-04 09:11:01,400 44k INFO Train Epoch: 1629 [52%] 2023-09-04 09:11:01,400 44k INFO Losses: [2.3524110317230225, 2.635582447052002, 6.632061958312988, 17.09296417236328, 0.5190930962562561], step: 34200, lr: 8.158598266746396e-05, reference_loss: 29.232112884521484 2023-09-04 09:11:07,722 44k INFO ====> Epoch: 1629, cost 14.61 s 2023-09-04 09:11:21,755 44k INFO ====> Epoch: 1630, cost 14.03 s 2023-09-04 09:11:35,837 44k INFO ====> Epoch: 1631, cost 14.08 s 2023-09-04 09:11:50,007 44k INFO ====> Epoch: 1632, cost 14.17 s 2023-09-04 09:12:04,412 44k INFO ====> Epoch: 1633, cost 14.41 s 2023-09-04 09:12:18,654 44k INFO ====> Epoch: 1634, cost 14.24 s 2023-09-04 09:12:32,803 44k INFO ====> Epoch: 1635, cost 14.15 s 2023-09-04 09:12:47,146 44k INFO ====> Epoch: 1636, cost 14.34 s 2023-09-04 09:13:01,397 44k INFO ====> Epoch: 1637, cost 14.25 s 2023-09-04 09:13:15,544 44k INFO ====> Epoch: 1638, cost 14.15 s 2023-09-04 09:13:16,931 44k INFO Train Epoch: 1639 [5%] 2023-09-04 09:13:16,932 44k INFO Losses: [2.5109331607818604, 2.7156283855438232, 6.671509742736816, 15.733382225036621, 0.6705777645111084], step: 34400, lr: 8.148405753515612e-05, reference_loss: 28.302032470703125 2023-09-04 09:13:21,580 44k INFO Saving model and optimizer state at iteration 1639 to ./logs\44k\G_34400.pth 2023-09-04 09:13:22,173 44k INFO Saving model and optimizer state at iteration 1639 to ./logs\44k\D_34400.pth 2023-09-04 09:13:23,328 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_32000.pth 2023-09-04 09:13:23,363 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_32000.pth 2023-09-04 09:13:36,042 44k INFO ====> Epoch: 1639, cost 20.50 s 2023-09-04 09:13:50,331 44k INFO ====> Epoch: 1640, cost 14.29 s 2023-09-04 09:14:04,658 44k INFO ====> Epoch: 1641, cost 14.33 s 2023-09-04 09:14:18,936 44k INFO ====> Epoch: 1642, cost 14.28 s 2023-09-04 09:14:33,167 44k INFO ====> Epoch: 1643, cost 14.23 s 2023-09-04 09:14:47,493 44k INFO ====> Epoch: 1644, cost 14.32 s 2023-09-04 09:15:01,764 44k INFO ====> Epoch: 1645, cost 14.27 s 2023-09-04 09:15:15,961 44k INFO ====> Epoch: 1646, cost 14.20 s 2023-09-04 09:15:29,967 44k INFO ====> Epoch: 1647, cost 14.01 s 2023-09-04 09:15:38,816 44k INFO Train Epoch: 1648 [57%] 2023-09-04 09:15:38,817 44k INFO Losses: [2.371063232421875, 2.4548745155334473, 6.26164436340332, 14.324589729309082, 0.652380645275116], step: 34600, lr: 8.139243379184544e-05, reference_loss: 26.064550399780273 2023-09-04 09:15:44,351 44k INFO ====> Epoch: 1648, cost 14.38 s 2023-09-04 09:15:58,619 44k INFO ====> Epoch: 1649, cost 14.27 s 2023-09-04 09:16:12,681 44k INFO ====> Epoch: 1650, cost 14.06 s 2023-09-04 09:16:26,762 44k INFO ====> Epoch: 1651, cost 14.08 s 2023-09-04 09:16:40,879 44k INFO ====> Epoch: 1652, cost 14.12 s 2023-09-04 09:16:55,122 44k INFO ====> Epoch: 1653, cost 14.24 s 2023-09-04 09:17:09,467 44k INFO ====> Epoch: 1654, cost 14.35 s 2023-09-04 09:17:23,648 44k INFO ====> Epoch: 1655, cost 14.18 s 2023-09-04 09:17:37,491 44k INFO ====> Epoch: 1656, cost 13.84 s 2023-09-04 09:17:51,890 44k INFO ====> Epoch: 1657, cost 14.40 s 2023-09-04 09:17:54,001 44k INFO Train Epoch: 1658 [10%] 2023-09-04 09:17:54,002 44k INFO Losses: [2.50719952583313, 2.252178907394409, 5.827363014221191, 16.098461151123047, 0.6025928854942322], step: 34800, lr: 8.129075045958844e-05, reference_loss: 27.287796020507812 2023-09-04 09:18:06,725 44k INFO ====> Epoch: 1658, cost 14.83 s 2023-09-04 09:18:20,818 44k INFO ====> Epoch: 1659, cost 14.09 s 2023-09-04 09:18:34,887 44k INFO ====> Epoch: 1660, cost 14.07 s 2023-09-04 09:18:49,206 44k INFO ====> Epoch: 1661, cost 14.32 s 2023-09-04 09:19:03,599 44k INFO ====> Epoch: 1662, cost 14.39 s 2023-09-04 09:19:17,684 44k INFO ====> Epoch: 1663, cost 14.08 s 2023-09-04 09:19:31,598 44k INFO ====> Epoch: 1664, cost 13.91 s 2023-09-04 09:19:45,745 44k INFO ====> Epoch: 1665, cost 14.15 s 2023-09-04 09:20:00,232 44k INFO ====> Epoch: 1666, cost 14.49 s 2023-09-04 09:20:09,960 44k INFO Train Epoch: 1667 [62%] 2023-09-04 09:20:09,960 44k INFO Losses: [2.2779016494750977, 2.520740270614624, 7.2603535652160645, 15.80125904083252, 0.5772539377212524], step: 35000, lr: 8.119934407803426e-05, reference_loss: 28.437509536743164 2023-09-04 09:20:14,854 44k INFO ====> Epoch: 1667, cost 14.62 s 2023-09-04 09:20:29,057 44k INFO ====> Epoch: 1668, cost 14.20 s 2023-09-04 09:20:43,214 44k INFO ====> Epoch: 1669, cost 14.16 s 2023-09-04 09:20:57,324 44k INFO ====> Epoch: 1670, cost 14.11 s 2023-09-04 09:21:11,567 44k INFO ====> Epoch: 1671, cost 14.24 s 2023-09-04 09:21:25,611 44k INFO ====> Epoch: 1672, cost 14.04 s 2023-09-04 09:21:39,568 44k INFO ====> Epoch: 1673, cost 13.96 s 2023-09-04 09:21:53,652 44k INFO ====> Epoch: 1674, cost 14.08 s 2023-09-04 09:22:07,778 44k INFO ====> Epoch: 1675, cost 14.13 s 2023-09-04 09:22:21,816 44k INFO ====> Epoch: 1676, cost 14.04 s 2023-09-04 09:22:24,624 44k INFO Train Epoch: 1677 [14%] 2023-09-04 09:22:24,625 44k INFO Losses: [2.3487138748168945, 2.7008440494537354, 8.338446617126465, 17.626079559326172, 0.7138145565986633], step: 35200, lr: 8.109790197219855e-05, reference_loss: 31.72789764404297 2023-09-04 09:22:29,305 44k INFO Saving model and optimizer state at iteration 1677 to ./logs\44k\G_35200.pth 2023-09-04 09:22:30,475 44k INFO Saving model and optimizer state at iteration 1677 to ./logs\44k\D_35200.pth 2023-09-04 09:22:31,542 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_32800.pth 2023-09-04 09:22:31,574 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_32800.pth 2023-09-04 09:22:43,027 44k INFO ====> Epoch: 1677, cost 21.21 s 2023-09-04 09:22:57,196 44k INFO ====> Epoch: 1678, cost 14.17 s 2023-09-04 09:23:11,566 44k INFO ====> Epoch: 1679, cost 14.37 s 2023-09-04 09:23:25,535 44k INFO ====> Epoch: 1680, cost 13.97 s 2023-09-04 09:23:39,608 44k INFO ====> Epoch: 1681, cost 14.07 s 2023-09-04 09:23:53,841 44k INFO ====> Epoch: 1682, cost 14.23 s 2023-09-04 09:24:08,248 44k INFO ====> Epoch: 1683, cost 14.41 s 2023-09-04 09:24:22,454 44k INFO ====> Epoch: 1684, cost 14.21 s 2023-09-04 09:24:36,574 44k INFO ====> Epoch: 1685, cost 14.12 s 2023-09-04 09:24:46,967 44k INFO Train Epoch: 1686 [67%] 2023-09-04 09:24:46,967 44k INFO Losses: [2.405470848083496, 2.23707914352417, 7.1465020179748535, 15.995467185974121, 0.6069339513778687], step: 35400, lr: 8.100671243674704e-05, reference_loss: 28.39145278930664 2023-09-04 09:24:51,131 44k INFO ====> Epoch: 1686, cost 14.56 s 2023-09-04 09:25:05,277 44k INFO ====> Epoch: 1687, cost 14.15 s 2023-09-04 09:25:19,279 44k INFO ====> Epoch: 1688, cost 14.00 s 2023-09-04 09:25:33,271 44k INFO ====> Epoch: 1689, cost 13.99 s 2023-09-04 09:25:47,434 44k INFO ====> Epoch: 1690, cost 14.16 s 2023-09-04 09:26:01,720 44k INFO ====> Epoch: 1691, cost 14.29 s 2023-09-04 09:26:16,028 44k INFO ====> Epoch: 1692, cost 14.31 s 2023-09-04 09:26:30,116 44k INFO ====> Epoch: 1693, cost 14.09 s 2023-09-04 09:26:44,270 44k INFO ====> Epoch: 1694, cost 14.15 s 2023-09-04 09:26:58,562 44k INFO ====> Epoch: 1695, cost 14.29 s 2023-09-04 09:27:01,946 44k INFO Train Epoch: 1696 [19%] 2023-09-04 09:27:01,947 44k INFO Losses: [2.4922780990600586, 2.296905279159546, 6.855761528015137, 16.564136505126953, 0.593026340007782], step: 35600, lr: 8.090551098506395e-05, reference_loss: 28.802108764648438 2023-09-04 09:27:12,821 44k INFO ====> Epoch: 1696, cost 14.26 s 2023-09-04 09:27:26,742 44k INFO ====> Epoch: 1697, cost 13.92 s 2023-09-04 09:27:40,627 44k INFO ====> Epoch: 1698, cost 13.88 s 2023-09-04 09:27:54,816 44k INFO ====> Epoch: 1699, cost 14.19 s 2023-09-04 09:28:08,912 44k INFO ====> Epoch: 1700, cost 14.10 s 2023-09-04 09:28:23,008 44k INFO ====> Epoch: 1701, cost 14.10 s 2023-09-04 09:28:36,962 44k INFO ====> Epoch: 1702, cost 13.95 s 2023-09-04 09:28:51,215 44k INFO ====> Epoch: 1703, cost 14.25 s 2023-09-04 09:29:05,615 44k INFO ====> Epoch: 1704, cost 14.40 s 2023-09-04 09:29:16,467 44k INFO Train Epoch: 1705 [71%] 2023-09-04 09:29:16,467 44k INFO Losses: [2.2611007690429688, 2.7388181686401367, 7.069143772125244, 15.308146476745605, 0.45513883233070374], step: 35800, lr: 8.081453778128458e-05, reference_loss: 27.832347869873047 2023-09-04 09:29:19,843 44k INFO ====> Epoch: 1705, cost 14.23 s 2023-09-04 09:29:33,833 44k INFO ====> Epoch: 1706, cost 13.99 s 2023-09-04 09:29:47,992 44k INFO ====> Epoch: 1707, cost 14.16 s 2023-09-04 09:30:02,407 44k INFO ====> Epoch: 1708, cost 14.41 s 2023-09-04 09:30:16,673 44k INFO ====> Epoch: 1709, cost 14.27 s 2023-09-04 09:30:30,722 44k INFO ====> Epoch: 1710, cost 14.05 s 2023-09-04 09:30:44,975 44k INFO ====> Epoch: 1711, cost 14.25 s 2023-09-04 09:30:59,315 44k INFO ====> Epoch: 1712, cost 14.34 s 2023-09-04 09:31:13,546 44k INFO ====> Epoch: 1713, cost 14.23 s 2023-09-04 09:31:27,483 44k INFO ====> Epoch: 1714, cost 13.94 s 2023-09-04 09:31:31,537 44k INFO Train Epoch: 1715 [24%] 2023-09-04 09:31:31,538 44k INFO Losses: [2.4039194583892822, 2.7121684551239014, 6.796450138092041, 15.348493576049805, 0.7990522980690002], step: 36000, lr: 8.071357641284309e-05, reference_loss: 28.06008529663086 2023-09-04 09:31:36,373 44k INFO Saving model and optimizer state at iteration 1715 to ./logs\44k\G_36000.pth 2023-09-04 09:31:36,964 44k INFO Saving model and optimizer state at iteration 1715 to ./logs\44k\D_36000.pth 2023-09-04 09:31:38,078 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_33600.pth 2023-09-04 09:31:38,112 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_33600.pth 2023-09-04 09:31:48,331 44k INFO ====> Epoch: 1715, cost 20.85 s 2023-09-04 09:32:02,679 44k INFO ====> Epoch: 1716, cost 14.35 s 2023-09-04 09:32:16,764 44k INFO ====> Epoch: 1717, cost 14.08 s 2023-09-04 09:32:30,829 44k INFO ====> Epoch: 1718, cost 14.07 s 2023-09-04 09:32:44,938 44k INFO ====> Epoch: 1719, cost 14.11 s 2023-09-04 09:32:59,212 44k INFO ====> Epoch: 1720, cost 14.27 s 2023-09-04 09:33:13,268 44k INFO ====> Epoch: 1721, cost 14.06 s 2023-09-04 09:33:27,099 44k INFO ====> Epoch: 1722, cost 13.83 s 2023-09-04 09:33:41,210 44k INFO ====> Epoch: 1723, cost 14.11 s 2023-09-04 09:33:52,871 44k INFO Train Epoch: 1724 [76%] 2023-09-04 09:33:52,871 44k INFO Losses: [2.264638662338257, 2.4475457668304443, 7.400599479675293, 17.388816833496094, 0.7952607274055481], step: 36200, lr: 8.062281902752576e-05, reference_loss: 30.29686164855957 2023-09-04 09:33:55,649 44k INFO ====> Epoch: 1724, cost 14.44 s 2023-09-04 09:34:09,866 44k INFO ====> Epoch: 1725, cost 14.22 s 2023-09-04 09:34:23,958 44k INFO ====> Epoch: 1726, cost 14.09 s 2023-09-04 09:34:37,987 44k INFO ====> Epoch: 1727, cost 14.03 s 2023-09-04 09:34:52,243 44k INFO ====> Epoch: 1728, cost 14.26 s 2023-09-04 09:35:06,569 44k INFO ====> Epoch: 1729, cost 14.33 s 2023-09-04 09:35:20,673 44k INFO ====> Epoch: 1730, cost 14.10 s 2023-09-04 09:35:34,609 44k INFO ====> Epoch: 1731, cost 13.94 s 2023-09-04 09:35:48,701 44k INFO ====> Epoch: 1732, cost 14.09 s 2023-09-04 09:36:03,021 44k INFO ====> Epoch: 1733, cost 14.32 s 2023-09-04 09:36:07,861 44k INFO Train Epoch: 1734 [29%] 2023-09-04 09:36:07,861 44k INFO Losses: [2.427746057510376, 2.4102282524108887, 6.419190883636475, 15.912064552307129, 0.5861032009124756], step: 36400, lr: 8.052209717276913e-05, reference_loss: 27.755334854125977 2023-09-04 09:36:17,441 44k INFO ====> Epoch: 1734, cost 14.42 s 2023-09-04 09:36:31,670 44k INFO ====> Epoch: 1735, cost 14.23 s 2023-09-04 09:36:45,857 44k INFO ====> Epoch: 1736, cost 14.19 s 2023-09-04 09:37:00,200 44k INFO ====> Epoch: 1737, cost 14.34 s 2023-09-04 09:37:14,320 44k INFO ====> Epoch: 1738, cost 14.12 s 2023-09-04 09:37:28,245 44k INFO ====> Epoch: 1739, cost 13.93 s 2023-09-04 09:37:42,273 44k INFO ====> Epoch: 1740, cost 14.03 s 2023-09-04 09:37:56,563 44k INFO ====> Epoch: 1741, cost 14.29 s 2023-09-04 09:38:10,849 44k INFO ====> Epoch: 1742, cost 14.29 s 2023-09-04 09:38:23,004 44k INFO Train Epoch: 1743 [81%] 2023-09-04 09:38:23,004 44k INFO Losses: [2.3727688789367676, 2.596980094909668, 7.310827255249023, 15.983464241027832, 0.790528416633606], step: 36600, lr: 8.043155509392122e-05, reference_loss: 29.0545711517334 2023-09-04 09:38:25,070 44k INFO ====> Epoch: 1743, cost 14.22 s 2023-09-04 09:38:39,330 44k INFO ====> Epoch: 1744, cost 14.26 s 2023-09-04 09:38:53,574 44k INFO ====> Epoch: 1745, cost 14.24 s 2023-09-04 09:39:07,851 44k INFO ====> Epoch: 1746, cost 14.28 s 2023-09-04 09:39:21,900 44k INFO ====> Epoch: 1747, cost 14.05 s 2023-09-04 09:39:35,892 44k INFO ====> Epoch: 1748, cost 13.99 s 2023-09-04 09:39:50,009 44k INFO ====> Epoch: 1749, cost 14.12 s 2023-09-04 09:40:04,245 44k INFO ====> Epoch: 1750, cost 14.24 s 2023-09-04 09:40:18,509 44k INFO ====> Epoch: 1751, cost 14.26 s 2023-09-04 09:40:32,535 44k INFO ====> Epoch: 1752, cost 14.03 s 2023-09-04 09:40:37,907 44k INFO Train Epoch: 1753 [33%] 2023-09-04 09:40:37,908 44k INFO Losses: [2.318830728530884, 2.5601139068603516, 6.699596405029297, 16.84906578063965, 0.6758157014846802], step: 36800, lr: 8.033107218464394e-05, reference_loss: 29.103422164916992 2023-09-04 09:40:42,606 44k INFO Saving model and optimizer state at iteration 1753 to ./logs\44k\G_36800.pth 2023-09-04 09:40:43,256 44k INFO Saving model and optimizer state at iteration 1753 to ./logs\44k\D_36800.pth 2023-09-04 09:40:44,486 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_34400.pth 2023-09-04 09:40:44,520 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_34400.pth 2023-09-04 09:40:53,231 44k INFO ====> Epoch: 1753, cost 20.70 s 2023-09-04 09:41:07,634 44k INFO ====> Epoch: 1754, cost 14.40 s 2023-09-04 09:41:21,724 44k INFO ====> Epoch: 1755, cost 14.09 s 2023-09-04 09:41:35,668 44k INFO ====> Epoch: 1756, cost 13.94 s 2023-09-04 09:41:50,029 44k INFO ====> Epoch: 1757, cost 14.36 s 2023-09-04 09:42:04,255 44k INFO ====> Epoch: 1758, cost 14.23 s 2023-09-04 09:42:18,626 44k INFO ====> Epoch: 1759, cost 14.37 s 2023-09-04 09:42:32,680 44k INFO ====> Epoch: 1760, cost 14.05 s 2023-09-04 09:42:46,886 44k INFO ====> Epoch: 1761, cost 14.21 s 2023-09-04 09:43:00,228 44k INFO Train Epoch: 1762 [86%] 2023-09-04 09:43:00,228 44k INFO Losses: [2.3799428939819336, 2.5192136764526367, 7.851846218109131, 16.02934455871582, 0.8123840093612671], step: 37000, lr: 8.024074490148745e-05, reference_loss: 29.592731475830078 2023-09-04 09:43:01,588 44k INFO ====> Epoch: 1762, cost 14.70 s 2023-09-04 09:43:15,645 44k INFO ====> Epoch: 1763, cost 14.06 s 2023-09-04 09:43:29,695 44k INFO ====> Epoch: 1764, cost 14.05 s 2023-09-04 09:43:43,642 44k INFO ====> Epoch: 1765, cost 13.95 s 2023-09-04 09:43:57,935 44k INFO ====> Epoch: 1766, cost 14.29 s 2023-09-04 09:44:12,062 44k INFO ====> Epoch: 1767, cost 14.13 s 2023-09-04 09:44:26,088 44k INFO ====> Epoch: 1768, cost 14.03 s 2023-09-04 09:44:40,195 44k INFO ====> Epoch: 1769, cost 14.11 s 2023-09-04 09:44:54,427 44k INFO ====> Epoch: 1770, cost 14.23 s 2023-09-04 09:45:08,628 44k INFO ====> Epoch: 1771, cost 14.20 s 2023-09-04 09:45:14,757 44k INFO Train Epoch: 1772 [38%] 2023-09-04 09:45:14,757 44k INFO Losses: [2.3415112495422363, 2.4562840461730957, 7.861859321594238, 17.133256912231445, 0.641074001789093], step: 37200, lr: 8.014050037083201e-05, reference_loss: 30.433984756469727 2023-09-04 09:45:22,876 44k INFO ====> Epoch: 1772, cost 14.25 s 2023-09-04 09:45:36,930 44k INFO ====> Epoch: 1773, cost 14.05 s 2023-09-04 09:45:51,069 44k INFO ====> Epoch: 1774, cost 14.14 s 2023-09-04 09:46:05,256 44k INFO ====> Epoch: 1775, cost 14.19 s 2023-09-04 09:46:19,370 44k INFO ====> Epoch: 1776, cost 14.11 s 2023-09-04 09:46:33,362 44k INFO ====> Epoch: 1777, cost 13.99 s 2023-09-04 09:46:47,567 44k INFO ====> Epoch: 1778, cost 14.21 s 2023-09-04 09:47:01,662 44k INFO ====> Epoch: 1779, cost 14.09 s 2023-09-04 09:47:15,741 44k INFO ====> Epoch: 1780, cost 14.08 s 2023-09-04 09:47:29,419 44k INFO Train Epoch: 1781 [90%] 2023-09-04 09:47:29,420 44k INFO Losses: [2.483525276184082, 2.3971269130706787, 6.6603617668151855, 16.966354370117188, 0.5052806735038757], step: 37400, lr: 8.005038737380067e-05, reference_loss: 29.012649536132812 2023-09-04 09:47:30,053 44k INFO ====> Epoch: 1781, cost 14.31 s 2023-09-04 09:47:44,215 44k INFO ====> Epoch: 1782, cost 14.16 s 2023-09-04 09:47:58,507 44k INFO ====> Epoch: 1783, cost 14.29 s 2023-09-04 09:48:12,783 44k INFO ====> Epoch: 1784, cost 14.28 s 2023-09-04 09:48:26,836 44k INFO ====> Epoch: 1785, cost 14.05 s 2023-09-04 09:48:40,895 44k INFO ====> Epoch: 1786, cost 14.06 s 2023-09-04 09:48:55,324 44k INFO ====> Epoch: 1787, cost 14.43 s 2023-09-04 09:49:09,619 44k INFO ====> Epoch: 1788, cost 14.30 s 2023-09-04 09:49:23,604 44k INFO ====> Epoch: 1789, cost 13.98 s 2023-09-04 09:49:37,640 44k INFO ====> Epoch: 1790, cost 14.04 s 2023-09-04 09:49:44,516 44k INFO Train Epoch: 1791 [43%] 2023-09-04 09:49:44,516 44k INFO Losses: [2.279127597808838, 2.5178260803222656, 6.422967910766602, 15.780908584594727, 0.764660656452179], step: 37600, lr: 7.995038065625431e-05, reference_loss: 27.76548957824707 2023-09-04 09:49:49,207 44k INFO Saving model and optimizer state at iteration 1791 to ./logs\44k\G_37600.pth 2023-09-04 09:49:49,804 44k INFO Saving model and optimizer state at iteration 1791 to ./logs\44k\D_37600.pth 2023-09-04 09:49:51,391 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_35200.pth 2023-09-04 09:49:51,425 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_35200.pth 2023-09-04 09:49:58,711 44k INFO ====> Epoch: 1791, cost 21.07 s 2023-09-04 09:50:12,969 44k INFO ====> Epoch: 1792, cost 14.26 s 2023-09-04 09:50:27,190 44k INFO ====> Epoch: 1793, cost 14.22 s 2023-09-04 09:50:41,118 44k INFO ====> Epoch: 1794, cost 13.93 s 2023-09-04 09:50:55,213 44k INFO ====> Epoch: 1795, cost 14.09 s 2023-09-04 09:51:09,491 44k INFO ====> Epoch: 1796, cost 14.28 s 2023-09-04 09:51:23,524 44k INFO ====> Epoch: 1797, cost 14.03 s 2023-09-04 09:51:37,502 44k INFO ====> Epoch: 1798, cost 13.98 s 2023-09-04 09:51:51,517 44k INFO ====> Epoch: 1799, cost 14.01 s 2023-09-04 09:52:05,663 44k INFO Train Epoch: 1800 [95%] 2023-09-04 09:52:05,663 44k INFO Losses: [2.5902318954467773, 2.322995901107788, 6.41019868850708, 15.789291381835938, 0.6844500303268433], step: 37800, lr: 7.986048143699072e-05, reference_loss: 27.797168731689453 2023-09-04 09:52:05,913 44k INFO ====> Epoch: 1800, cost 14.40 s 2023-09-04 09:52:19,900 44k INFO ====> Epoch: 1801, cost 13.99 s 2023-09-04 09:52:34,124 44k INFO ====> Epoch: 1802, cost 14.22 s 2023-09-04 09:52:48,231 44k INFO ====> Epoch: 1803, cost 14.11 s 2023-09-04 09:53:02,470 44k INFO ====> Epoch: 1804, cost 14.24 s 2023-09-04 09:53:16,552 44k INFO ====> Epoch: 1805, cost 14.08 s 2023-09-04 09:53:30,571 44k INFO ====> Epoch: 1806, cost 14.02 s 2023-09-04 09:53:44,800 44k INFO ====> Epoch: 1807, cost 14.23 s 2023-09-04 09:53:59,131 44k INFO ====> Epoch: 1808, cost 14.33 s 2023-09-04 09:54:13,572 44k INFO ====> Epoch: 1809, cost 14.44 s 2023-09-04 09:54:21,141 44k INFO Train Epoch: 1810 [48%] 2023-09-04 09:54:21,141 44k INFO Losses: [2.3281517028808594, 2.4855735301971436, 8.074657440185547, 17.771257400512695, 0.7005745768547058], step: 38000, lr: 7.976071196838225e-05, reference_loss: 31.36021614074707 2023-09-04 09:54:27,865 44k INFO ====> Epoch: 1810, cost 14.29 s 2023-09-04 09:54:42,023 44k INFO ====> Epoch: 1811, cost 14.16 s 2023-09-04 09:54:56,417 44k INFO ====> Epoch: 1812, cost 14.39 s 2023-09-04 09:55:10,786 44k INFO ====> Epoch: 1813, cost 14.37 s 2023-09-04 09:55:24,825 44k INFO ====> Epoch: 1814, cost 14.04 s 2023-09-04 09:55:38,806 44k INFO ====> Epoch: 1815, cost 13.98 s 2023-09-04 09:55:53,183 44k INFO ====> Epoch: 1816, cost 14.38 s 2023-09-04 09:56:07,421 44k INFO ====> Epoch: 1817, cost 14.24 s 2023-09-04 09:56:21,465 44k INFO ====> Epoch: 1818, cost 14.05 s 2023-09-04 09:56:35,644 44k INFO ====> Epoch: 1819, cost 14.18 s 2023-09-04 09:56:36,340 44k INFO Train Epoch: 1820 [0%] 2023-09-04 09:56:36,340 44k INFO Losses: [2.426288366317749, 2.506497383117676, 7.29753303527832, 16.034622192382812, 0.772846519947052], step: 38200, lr: 7.96610671414825e-05, reference_loss: 29.03778839111328 2023-09-04 09:56:50,103 44k INFO ====> Epoch: 1820, cost 14.46 s 2023-09-04 09:57:04,362 44k INFO ====> Epoch: 1821, cost 14.26 s 2023-09-04 09:57:18,441 44k INFO ====> Epoch: 1822, cost 14.08 s 2023-09-04 09:57:32,479 44k INFO ====> Epoch: 1823, cost 14.04 s 2023-09-04 09:57:46,557 44k INFO ====> Epoch: 1824, cost 14.08 s 2023-09-04 09:58:00,745 44k INFO ====> Epoch: 1825, cost 14.19 s 2023-09-04 09:58:14,956 44k INFO ====> Epoch: 1826, cost 14.21 s 2023-09-04 09:58:28,931 44k INFO ====> Epoch: 1827, cost 13.98 s 2023-09-04 09:58:43,034 44k INFO ====> Epoch: 1828, cost 14.10 s 2023-09-04 09:58:51,265 44k INFO Train Epoch: 1829 [52%] 2023-09-04 09:58:51,265 44k INFO Losses: [2.557300090789795, 2.3469200134277344, 7.233905792236328, 17.053682327270508, 0.7121421098709106], step: 38400, lr: 7.95714932372316e-05, reference_loss: 29.90395164489746 2023-09-04 09:58:55,957 44k INFO Saving model and optimizer state at iteration 1829 to ./logs\44k\G_38400.pth 2023-09-04 09:58:56,647 44k INFO Saving model and optimizer state at iteration 1829 to ./logs\44k\D_38400.pth 2023-09-04 09:58:57,713 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_36000.pth 2023-09-04 09:58:57,750 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_36000.pth 2023-09-04 09:59:03,785 44k INFO ====> Epoch: 1829, cost 20.75 s 2023-09-04 09:59:17,931 44k INFO ====> Epoch: 1830, cost 14.15 s 2023-09-04 09:59:31,781 44k INFO ====> Epoch: 1831, cost 13.85 s 2023-09-04 09:59:45,854 44k INFO ====> Epoch: 1832, cost 14.07 s 2023-09-04 10:00:00,262 44k INFO ====> Epoch: 1833, cost 14.41 s 2023-09-04 10:00:14,633 44k INFO ====> Epoch: 1834, cost 14.37 s 2023-09-04 10:00:28,701 44k INFO ====> Epoch: 1835, cost 14.07 s 2023-09-04 10:00:42,871 44k INFO ====> Epoch: 1836, cost 14.17 s 2023-09-04 10:00:57,156 44k INFO ====> Epoch: 1837, cost 14.29 s 2023-09-04 10:01:11,344 44k INFO ====> Epoch: 1838, cost 14.19 s 2023-09-04 10:01:12,745 44k INFO Train Epoch: 1839 [5%] 2023-09-04 10:01:12,745 44k INFO Losses: [2.2928075790405273, 2.672114372253418, 6.360512733459473, 14.62458610534668, 0.7159813046455383], step: 38600, lr: 7.947208480074573e-05, reference_loss: 26.666000366210938 2023-09-04 10:01:25,589 44k INFO ====> Epoch: 1839, cost 14.25 s 2023-09-04 10:01:39,627 44k INFO ====> Epoch: 1840, cost 14.04 s 2023-09-04 10:01:53,970 44k INFO ====> Epoch: 1841, cost 14.34 s 2023-09-04 10:02:08,384 44k INFO ====> Epoch: 1842, cost 14.41 s 2023-09-04 10:02:22,466 44k INFO ====> Epoch: 1843, cost 14.08 s 2023-09-04 10:02:36,479 44k INFO ====> Epoch: 1844, cost 14.01 s 2023-09-04 10:02:50,776 44k INFO ====> Epoch: 1845, cost 14.30 s 2023-09-04 10:03:04,840 44k INFO ====> Epoch: 1846, cost 14.06 s 2023-09-04 10:03:18,848 44k INFO ====> Epoch: 1847, cost 14.01 s 2023-09-04 10:03:27,703 44k INFO Train Epoch: 1848 [57%] 2023-09-04 10:03:27,703 44k INFO Losses: [2.461353302001953, 2.280819892883301, 7.755100250244141, 17.652475357055664, 0.5870316028594971], step: 38800, lr: 7.938272339535662e-05, reference_loss: 30.73678207397461 2023-09-04 10:03:33,145 44k INFO ====> Epoch: 1848, cost 14.30 s 2023-09-04 10:03:47,121 44k INFO ====> Epoch: 1849, cost 13.98 s 2023-09-04 10:04:01,283 44k INFO ====> Epoch: 1850, cost 14.16 s 2023-09-04 10:04:15,659 44k INFO ====> Epoch: 1851, cost 14.38 s 2023-09-04 10:04:29,759 44k INFO ====> Epoch: 1852, cost 14.10 s 2023-09-04 10:04:43,692 44k INFO ====> Epoch: 1853, cost 13.93 s 2023-09-04 10:04:57,917 44k INFO ====> Epoch: 1854, cost 14.23 s 2023-09-04 10:05:12,082 44k INFO ====> Epoch: 1855, cost 14.16 s 2023-09-04 10:05:25,956 44k INFO ====> Epoch: 1856, cost 13.87 s 2023-09-04 10:05:39,971 44k INFO ====> Epoch: 1857, cost 14.01 s 2023-09-04 10:05:42,046 44k INFO Train Epoch: 1858 [10%] 2023-09-04 10:05:42,047 44k INFO Losses: [2.4259941577911377, 2.5503623485565186, 6.612611770629883, 15.61670207977295, 0.65012127161026], step: 39000, lr: 7.928355078848854e-05, reference_loss: 27.855791091918945 2023-09-04 10:05:54,609 44k INFO ====> Epoch: 1858, cost 14.64 s 2023-09-04 10:06:09,037 44k INFO ====> Epoch: 1859, cost 14.43 s 2023-09-04 10:06:23,124 44k INFO ====> Epoch: 1860, cost 14.09 s 2023-09-04 10:06:37,175 44k INFO ====> Epoch: 1861, cost 14.05 s 2023-09-04 10:06:51,465 44k INFO ====> Epoch: 1862, cost 14.29 s 2023-09-04 10:07:05,909 44k INFO ====> Epoch: 1863, cost 14.44 s 2023-09-04 10:07:19,927 44k INFO ====> Epoch: 1864, cost 14.02 s 2023-09-04 10:07:33,933 44k INFO ====> Epoch: 1865, cost 14.01 s 2023-09-04 10:07:48,205 44k INFO ====> Epoch: 1866, cost 14.27 s 2023-09-04 10:07:57,908 44k INFO Train Epoch: 1867 [62%] 2023-09-04 10:07:57,908 44k INFO Losses: [2.532710075378418, 2.1257550716400146, 5.493768692016602, 15.088891983032227, 0.6509276628494263], step: 39200, lr: 7.919440137784376e-05, reference_loss: 25.89205551147461 2023-09-04 10:08:02,640 44k INFO Saving model and optimizer state at iteration 1867 to ./logs\44k\G_39200.pth 2023-09-04 10:08:03,336 44k INFO Saving model and optimizer state at iteration 1867 to ./logs\44k\D_39200.pth 2023-09-04 10:08:04,447 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_36800.pth 2023-09-04 10:08:04,480 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_36800.pth 2023-09-04 10:08:09,100 44k INFO ====> Epoch: 1867, cost 20.89 s 2023-09-04 10:08:23,362 44k INFO ====> Epoch: 1868, cost 14.26 s 2023-09-04 10:08:37,523 44k INFO ====> Epoch: 1869, cost 14.16 s 2023-09-04 10:08:51,702 44k INFO ====> Epoch: 1870, cost 14.18 s 2023-09-04 10:09:06,031 44k INFO ====> Epoch: 1871, cost 14.33 s 2023-09-04 10:09:20,249 44k INFO ====> Epoch: 1872, cost 14.22 s 2023-09-04 10:09:34,133 44k INFO ====> Epoch: 1873, cost 13.88 s 2023-09-04 10:09:48,352 44k INFO ====> Epoch: 1874, cost 14.22 s 2023-09-04 10:10:02,438 44k INFO ====> Epoch: 1875, cost 14.09 s 2023-09-04 10:10:16,723 44k INFO ====> Epoch: 1876, cost 14.28 s 2023-09-04 10:10:19,452 44k INFO Train Epoch: 1877 [14%] 2023-09-04 10:10:19,452 44k INFO Losses: [2.3887572288513184, 2.29738450050354, 7.404932498931885, 16.945632934570312, 0.759294331073761], step: 39400, lr: 7.909546404112776e-05, reference_loss: 29.796001434326172 2023-09-04 10:10:30,958 44k INFO ====> Epoch: 1877, cost 14.23 s 2023-09-04 10:10:45,066 44k INFO ====> Epoch: 1878, cost 14.11 s 2023-09-04 10:10:59,382 44k INFO ====> Epoch: 1879, cost 14.32 s 2023-09-04 10:11:13,515 44k INFO ====> Epoch: 1880, cost 14.13 s 2023-09-04 10:11:27,478 44k INFO ====> Epoch: 1881, cost 13.96 s 2023-09-04 10:11:41,528 44k INFO ====> Epoch: 1882, cost 14.05 s 2023-09-04 10:11:55,966 44k INFO ====> Epoch: 1883, cost 14.44 s 2023-09-04 10:12:10,423 44k INFO ====> Epoch: 1884, cost 14.46 s 2023-09-04 10:12:24,665 44k INFO ====> Epoch: 1885, cost 14.24 s 2023-09-04 10:12:34,950 44k INFO Train Epoch: 1886 [67%] 2023-09-04 10:12:34,951 44k INFO Losses: [2.263406276702881, 2.6389074325561523, 8.475908279418945, 16.6494197845459, 0.5985537171363831], step: 39600, lr: 7.900652612230582e-05, reference_loss: 30.626195907592773 2023-09-04 10:12:39,073 44k INFO ====> Epoch: 1886, cost 14.41 s 2023-09-04 10:12:53,378 44k INFO ====> Epoch: 1887, cost 14.30 s 2023-09-04 10:13:07,726 44k INFO ====> Epoch: 1888, cost 14.35 s 2023-09-04 10:13:21,663 44k INFO ====> Epoch: 1889, cost 13.94 s 2023-09-04 10:13:35,716 44k INFO ====> Epoch: 1890, cost 14.05 s 2023-09-04 10:13:49,878 44k INFO ====> Epoch: 1891, cost 14.16 s 2023-09-04 10:14:04,264 44k INFO ====> Epoch: 1892, cost 14.39 s 2023-09-04 10:14:18,583 44k INFO ====> Epoch: 1893, cost 14.32 s 2023-09-04 10:14:32,753 44k INFO ====> Epoch: 1894, cost 14.17 s 2023-09-04 10:14:46,886 44k INFO ====> Epoch: 1895, cost 14.13 s 2023-09-04 10:14:50,274 44k INFO Train Epoch: 1896 [19%] 2023-09-04 10:14:50,275 44k INFO Losses: [2.382448673248291, 2.4594063758850098, 6.734246253967285, 16.712873458862305, 0.5644185543060303], step: 39800, lr: 7.890782349760348e-05, reference_loss: 28.8533935546875 2023-09-04 10:15:01,213 44k INFO ====> Epoch: 1896, cost 14.33 s 2023-09-04 10:15:15,309 44k INFO ====> Epoch: 1897, cost 14.10 s 2023-09-04 10:15:29,390 44k INFO ====> Epoch: 1898, cost 14.08 s 2023-09-04 10:15:43,460 44k INFO ====> Epoch: 1899, cost 14.07 s 2023-09-04 10:15:57,652 44k INFO ====> Epoch: 1900, cost 14.19 s 2023-09-04 10:16:11,823 44k INFO ====> Epoch: 1901, cost 14.17 s 2023-09-04 10:16:25,928 44k INFO ====> Epoch: 1902, cost 14.11 s 2023-09-04 10:16:39,874 44k INFO ====> Epoch: 1903, cost 13.95 s 2023-09-04 10:16:54,193 44k INFO ====> Epoch: 1904, cost 14.32 s 2023-09-04 10:17:05,254 44k INFO Train Epoch: 1905 [71%] 2023-09-04 10:17:05,255 44k INFO Losses: [2.2442376613616943, 2.8127126693725586, 7.810869216918945, 16.26656150817871, 0.6752636432647705], step: 40000, lr: 7.881909656887598e-05, reference_loss: 29.80964469909668 2023-09-04 10:17:10,016 44k INFO Saving model and optimizer state at iteration 1905 to ./logs\44k\G_40000.pth 2023-09-04 10:17:10,652 44k INFO Saving model and optimizer state at iteration 1905 to ./logs\44k\D_40000.pth 2023-09-04 10:17:11,741 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_37600.pth 2023-09-04 10:17:11,775 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_37600.pth 2023-09-04 10:17:14,955 44k INFO ====> Epoch: 1905, cost 20.76 s 2023-09-04 10:17:29,057 44k INFO ====> Epoch: 1906, cost 14.10 s 2023-09-04 10:17:43,004 44k INFO ====> Epoch: 1907, cost 13.95 s 2023-09-04 10:17:57,515 44k INFO ====> Epoch: 1908, cost 14.51 s 2023-09-04 10:18:11,892 44k INFO ====> Epoch: 1909, cost 14.38 s 2023-09-04 10:18:26,022 44k INFO ====> Epoch: 1910, cost 14.13 s 2023-09-04 10:18:40,237 44k INFO ====> Epoch: 1911, cost 14.21 s 2023-09-04 10:18:54,494 44k INFO ====> Epoch: 1912, cost 14.26 s 2023-09-04 10:19:08,775 44k INFO ====> Epoch: 1913, cost 14.28 s 2023-09-04 10:19:22,912 44k INFO ====> Epoch: 1914, cost 14.14 s 2023-09-04 10:19:26,990 44k INFO Train Epoch: 1915 [24%] 2023-09-04 10:19:26,990 44k INFO Losses: [2.3652610778808594, 2.5659053325653076, 5.54791784286499, 13.650594711303711, 0.746734619140625], step: 40200, lr: 7.872062809937293e-05, reference_loss: 24.876413345336914 2023-09-04 10:19:37,139 44k INFO ====> Epoch: 1915, cost 14.23 s 2023-09-04 10:19:51,311 44k INFO ====> Epoch: 1916, cost 14.17 s 2023-09-04 10:20:05,643 44k INFO ====> Epoch: 1917, cost 14.33 s 2023-09-04 10:20:19,912 44k INFO ====> Epoch: 1918, cost 14.27 s 2023-09-04 10:20:34,205 44k INFO ====> Epoch: 1919, cost 14.29 s 2023-09-04 10:20:48,478 44k INFO ====> Epoch: 1920, cost 14.27 s 2023-09-04 10:21:02,780 44k INFO ====> Epoch: 1921, cost 14.30 s 2023-09-04 10:21:16,868 44k INFO ====> Epoch: 1922, cost 14.09 s 2023-09-04 10:21:30,927 44k INFO ====> Epoch: 1923, cost 14.06 s 2023-09-04 10:21:42,475 44k INFO Train Epoch: 1924 [76%] 2023-09-04 10:21:42,475 44k INFO Losses: [2.278484582901001, 2.619935989379883, 6.964365482330322, 17.35150718688965, 0.6130533814430237], step: 40400, lr: 7.863211166020172e-05, reference_loss: 29.827346801757812 2023-09-04 10:21:45,247 44k INFO ====> Epoch: 1924, cost 14.32 s 2023-09-04 10:21:59,506 44k INFO ====> Epoch: 1925, cost 14.26 s 2023-09-04 10:22:13,610 44k INFO ====> Epoch: 1926, cost 14.10 s 2023-09-04 10:22:27,671 44k INFO ====> Epoch: 1927, cost 14.06 s 2023-09-04 10:22:41,755 44k INFO ====> Epoch: 1928, cost 14.08 s 2023-09-04 10:22:55,920 44k INFO ====> Epoch: 1929, cost 14.16 s 2023-09-04 10:23:10,094 44k INFO ====> Epoch: 1930, cost 14.17 s 2023-09-04 10:23:24,156 44k INFO ====> Epoch: 1931, cost 14.06 s 2023-09-04 10:23:38,079 44k INFO ====> Epoch: 1932, cost 13.92 s 2023-09-04 10:23:52,408 44k INFO ====> Epoch: 1933, cost 14.33 s 2023-09-04 10:23:57,313 44k INFO Train Epoch: 1934 [29%] 2023-09-04 10:23:57,313 44k INFO Losses: [2.3882882595062256, 2.4309346675872803, 7.000330924987793, 14.5903959274292, 0.5504056215286255], step: 40600, lr: 7.853387679040456e-05, reference_loss: 26.960355758666992 2023-09-04 10:24:07,085 44k INFO ====> Epoch: 1934, cost 14.68 s 2023-09-04 10:24:21,299 44k INFO ====> Epoch: 1935, cost 14.21 s 2023-09-04 10:24:35,438 44k INFO ====> Epoch: 1936, cost 14.14 s 2023-09-04 10:24:49,575 44k INFO ====> Epoch: 1937, cost 14.14 s 2023-09-04 10:25:04,013 44k INFO ====> Epoch: 1938, cost 14.44 s 2023-09-04 10:25:18,103 44k INFO ====> Epoch: 1939, cost 14.09 s 2023-09-04 10:25:32,117 44k INFO ====> Epoch: 1940, cost 14.01 s 2023-09-04 10:25:46,343 44k INFO ====> Epoch: 1941, cost 14.23 s 2023-09-04 10:26:00,633 44k INFO ====> Epoch: 1942, cost 14.29 s 2023-09-04 10:26:13,097 44k INFO Train Epoch: 1943 [81%] 2023-09-04 10:26:13,097 44k INFO Losses: [2.4857473373413086, 2.412482261657715, 6.4207563400268555, 15.506227493286133, 0.7217637896537781], step: 40800, lr: 7.844557034143897e-05, reference_loss: 27.546977996826172 2023-09-04 10:26:17,751 44k INFO Saving model and optimizer state at iteration 1943 to ./logs\44k\G_40800.pth 2023-09-04 10:26:18,934 44k INFO Saving model and optimizer state at iteration 1943 to ./logs\44k\D_40800.pth 2023-09-04 10:26:20,568 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_38400.pth 2023-09-04 10:26:20,602 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_38400.pth 2023-09-04 10:26:22,412 44k INFO ====> Epoch: 1943, cost 21.78 s 2023-09-04 10:26:36,582 44k INFO ====> Epoch: 1944, cost 14.17 s 2023-09-04 10:26:50,754 44k INFO ====> Epoch: 1945, cost 14.17 s 2023-09-04 10:27:04,872 44k INFO ====> Epoch: 1946, cost 14.12 s 2023-09-04 10:27:19,061 44k INFO ====> Epoch: 1947, cost 14.19 s 2023-09-04 10:27:32,958 44k INFO ====> Epoch: 1948, cost 13.90 s 2023-09-04 10:27:47,024 44k INFO ====> Epoch: 1949, cost 14.07 s 2023-09-04 10:28:01,223 44k INFO ====> Epoch: 1950, cost 14.20 s 2023-09-04 10:28:15,357 44k INFO ====> Epoch: 1951, cost 14.13 s 2023-09-04 10:28:29,468 44k INFO ====> Epoch: 1952, cost 14.11 s 2023-09-04 10:28:34,925 44k INFO Train Epoch: 1953 [33%] 2023-09-04 10:28:34,926 44k INFO Losses: [2.3718833923339844, 2.510206937789917, 6.666513919830322, 15.292576789855957, 0.6467474699020386], step: 41000, lr: 7.834756851717211e-05, reference_loss: 27.487930297851562 2023-09-04 10:28:43,874 44k INFO ====> Epoch: 1953, cost 14.41 s 2023-09-04 10:28:58,111 44k INFO ====> Epoch: 1954, cost 14.24 s 2023-09-04 10:29:12,385 44k INFO ====> Epoch: 1955, cost 14.27 s 2023-09-04 10:29:26,544 44k INFO ====> Epoch: 1956, cost 14.16 s 2023-09-04 10:29:40,496 44k INFO ====> Epoch: 1957, cost 13.95 s 2023-09-04 10:29:54,986 44k INFO ====> Epoch: 1958, cost 14.49 s 2023-09-04 10:30:09,233 44k INFO ====> Epoch: 1959, cost 14.25 s 2023-09-04 10:30:23,245 44k INFO ====> Epoch: 1960, cost 14.01 s 2023-09-04 10:30:37,340 44k INFO ====> Epoch: 1961, cost 14.09 s 2023-09-04 10:30:50,577 44k INFO Train Epoch: 1962 [86%] 2023-09-04 10:30:50,577 44k INFO Losses: [2.2031924724578857, 2.6655774116516113, 8.576887130737305, 15.418397903442383, 0.590086042881012], step: 41200, lr: 7.825947156024605e-05, reference_loss: 29.45414161682129 2023-09-04 10:30:51,931 44k INFO ====> Epoch: 1962, cost 14.59 s 2023-09-04 10:31:06,241 44k INFO ====> Epoch: 1963, cost 14.31 s 2023-09-04 10:31:20,258 44k INFO ====> Epoch: 1964, cost 14.02 s 2023-09-04 10:31:34,122 44k INFO ====> Epoch: 1965, cost 13.86 s 2023-09-04 10:31:48,172 44k INFO ====> Epoch: 1966, cost 14.05 s 2023-09-04 10:32:02,432 44k INFO ====> Epoch: 1967, cost 14.26 s 2023-09-04 10:32:16,687 44k INFO ====> Epoch: 1968, cost 14.26 s 2023-09-04 10:32:30,898 44k INFO ====> Epoch: 1969, cost 14.21 s 2023-09-04 10:32:45,049 44k INFO ====> Epoch: 1970, cost 14.15 s 2023-09-04 10:32:59,302 44k INFO ====> Epoch: 1971, cost 14.25 s 2023-09-04 10:33:05,508 44k INFO Train Epoch: 1972 [38%] 2023-09-04 10:33:05,509 44k INFO Losses: [2.4443113803863525, 2.587404251098633, 6.759256839752197, 16.1433048248291, 0.4864508807659149], step: 41400, lr: 7.81617022286486e-05, reference_loss: 28.420726776123047 2023-09-04 10:33:13,767 44k INFO ====> Epoch: 1972, cost 14.46 s 2023-09-04 10:33:27,859 44k INFO ====> Epoch: 1973, cost 14.09 s 2023-09-04 10:33:41,882 44k INFO ====> Epoch: 1974, cost 14.02 s 2023-09-04 10:33:56,106 44k INFO ====> Epoch: 1975, cost 14.22 s 2023-09-04 10:34:10,230 44k INFO ====> Epoch: 1976, cost 14.12 s 2023-09-04 10:34:24,333 44k INFO ====> Epoch: 1977, cost 14.10 s 2023-09-04 10:34:38,376 44k INFO ====> Epoch: 1978, cost 14.04 s 2023-09-04 10:34:52,602 44k INFO ====> Epoch: 1979, cost 14.23 s 2023-09-04 10:35:06,831 44k INFO ====> Epoch: 1980, cost 14.23 s 2023-09-04 10:35:20,474 44k INFO Train Epoch: 1981 [90%] 2023-09-04 10:35:20,475 44k INFO Losses: [2.332559585571289, 2.4819371700286865, 7.358489990234375, 16.686309814453125, 0.7930516600608826], step: 41600, lr: 7.807381426677785e-05, reference_loss: 29.652347564697266 2023-09-04 10:35:25,242 44k INFO Saving model and optimizer state at iteration 1981 to ./logs\44k\G_41600.pth 2023-09-04 10:35:25,870 44k INFO Saving model and optimizer state at iteration 1981 to ./logs\44k\D_41600.pth 2023-09-04 10:35:27,031 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_39200.pth 2023-09-04 10:35:27,064 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_39200.pth 2023-09-04 10:35:27,490 44k INFO ====> Epoch: 1981, cost 20.66 s 2023-09-04 10:35:41,548 44k INFO ====> Epoch: 1982, cost 14.06 s 2023-09-04 10:35:56,168 44k INFO ====> Epoch: 1983, cost 14.62 s 2023-09-04 10:36:10,581 44k INFO ====> Epoch: 1984, cost 14.41 s 2023-09-04 10:36:24,680 44k INFO ====> Epoch: 1985, cost 14.10 s 2023-09-04 10:36:38,703 44k INFO ====> Epoch: 1986, cost 14.02 s 2023-09-04 10:36:53,121 44k INFO ====> Epoch: 1987, cost 14.42 s 2023-09-04 10:37:07,652 44k INFO ====> Epoch: 1988, cost 14.53 s 2023-09-04 10:37:21,852 44k INFO ====> Epoch: 1989, cost 14.20 s 2023-09-04 10:37:35,860 44k INFO ====> Epoch: 1990, cost 14.01 s 2023-09-04 10:37:42,769 44k INFO Train Epoch: 1991 [43%] 2023-09-04 10:37:42,769 44k INFO Losses: [2.588327407836914, 2.0561046600341797, 6.311371803283691, 16.088788986206055, 0.5901093482971191], step: 41800, lr: 7.797627687630045e-05, reference_loss: 27.634702682495117 2023-09-04 10:37:50,389 44k INFO ====> Epoch: 1991, cost 14.53 s 2023-09-04 10:38:04,572 44k INFO ====> Epoch: 1992, cost 14.18 s 2023-09-04 10:38:18,785 44k INFO ====> Epoch: 1993, cost 14.21 s 2023-09-04 10:38:32,761 44k INFO ====> Epoch: 1994, cost 13.98 s 2023-09-04 10:38:46,847 44k INFO ====> Epoch: 1995, cost 14.09 s 2023-09-04 10:39:00,989 44k INFO ====> Epoch: 1996, cost 14.14 s 2023-09-04 10:39:15,142 44k INFO ====> Epoch: 1997, cost 14.15 s 2023-09-04 10:39:29,052 44k INFO ====> Epoch: 1998, cost 13.91 s 2023-09-04 10:39:43,083 44k INFO ====> Epoch: 1999, cost 14.03 s 2023-09-04 10:39:57,188 44k INFO Train Epoch: 2000 [95%] 2023-09-04 10:39:57,188 44k INFO Losses: [2.08919620513916, 2.744279623031616, 8.877264976501465, 16.776826858520508, 0.32421356439590454], step: 42000, lr: 7.788859741367973e-05, reference_loss: 30.81178092956543 2023-09-04 10:39:57,509 44k INFO ====> Epoch: 2000, cost 14.43 s 2023-09-04 10:40:11,686 44k INFO ====> Epoch: 2001, cost 14.18 s 2023-09-04 10:40:25,743 44k INFO ====> Epoch: 2002, cost 14.06 s 2023-09-04 10:40:39,786 44k INFO ====> Epoch: 2003, cost 14.04 s 2023-09-04 10:40:53,975 44k INFO ====> Epoch: 2004, cost 14.19 s 2023-09-04 10:41:08,167 44k INFO ====> Epoch: 2005, cost 14.19 s 2023-09-04 10:41:22,068 44k INFO ====> Epoch: 2006, cost 13.90 s 2023-09-04 10:41:36,076 44k INFO ====> Epoch: 2007, cost 14.01 s 2023-09-04 10:41:50,458 44k INFO ====> Epoch: 2008, cost 14.38 s 2023-09-04 10:42:04,889 44k INFO ====> Epoch: 2009, cost 14.43 s 2023-09-04 10:42:12,544 44k INFO Train Epoch: 2010 [48%] 2023-09-04 10:42:12,544 44k INFO Losses: [2.5352940559387207, 2.2310173511505127, 6.986213207244873, 15.992874145507812, 0.6230218410491943], step: 42200, lr: 7.779129141408152e-05, reference_loss: 28.36842155456543 2023-09-04 10:42:19,365 44k INFO ====> Epoch: 2010, cost 14.48 s 2023-09-04 10:42:33,530 44k INFO ====> Epoch: 2011, cost 14.16 s 2023-09-04 10:42:47,926 44k INFO ====> Epoch: 2012, cost 14.40 s 2023-09-04 10:43:02,204 44k INFO ====> Epoch: 2013, cost 14.28 s 2023-09-04 10:43:16,278 44k INFO ====> Epoch: 2014, cost 14.07 s 2023-09-04 10:43:30,170 44k INFO ====> Epoch: 2015, cost 13.89 s 2023-09-04 10:43:44,288 44k INFO ====> Epoch: 2016, cost 14.12 s 2023-09-04 10:43:58,572 44k INFO ====> Epoch: 2017, cost 14.28 s 2023-09-04 10:44:12,693 44k INFO ====> Epoch: 2018, cost 14.12 s 2023-09-04 10:44:26,895 44k INFO ====> Epoch: 2019, cost 14.20 s 2023-09-04 10:44:27,603 44k INFO Train Epoch: 2020 [0%] 2023-09-04 10:44:27,603 44k INFO Losses: [2.3638429641723633, 2.5347487926483154, 7.117061614990234, 15.131765365600586, 0.6551921367645264], step: 42400, lr: 7.76941069785873e-05, reference_loss: 27.802610397338867 2023-09-04 10:44:32,264 44k INFO Saving model and optimizer state at iteration 2020 to ./logs\44k\G_42400.pth 2023-09-04 10:44:32,811 44k INFO Saving model and optimizer state at iteration 2020 to ./logs\44k\D_42400.pth 2023-09-04 10:44:33,890 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_40000.pth 2023-09-04 10:44:33,923 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_40000.pth 2023-09-04 10:44:47,307 44k INFO ====> Epoch: 2020, cost 20.41 s 2023-09-04 10:45:01,459 44k INFO ====> Epoch: 2021, cost 14.15 s 2023-09-04 10:45:15,683 44k INFO ====> Epoch: 2022, cost 14.23 s 2023-09-04 10:45:29,662 44k INFO ====> Epoch: 2023, cost 13.98 s 2023-09-04 10:45:43,754 44k INFO ====> Epoch: 2024, cost 14.09 s 2023-09-04 10:45:57,959 44k INFO ====> Epoch: 2025, cost 14.21 s 2023-09-04 10:46:12,093 44k INFO ====> Epoch: 2026, cost 14.13 s 2023-09-04 10:46:26,244 44k INFO ====> Epoch: 2027, cost 14.15 s 2023-09-04 10:46:40,303 44k INFO ====> Epoch: 2028, cost 14.06 s 2023-09-04 10:46:48,532 44k INFO Train Epoch: 2029 [52%] 2023-09-04 10:46:48,532 44k INFO Losses: [2.589383840560913, 2.2425267696380615, 6.574981212615967, 16.11432647705078, 0.6609695553779602], step: 42600, lr: 7.760674479842725e-05, reference_loss: 28.182186126708984 2023-09-04 10:46:54,685 44k INFO ====> Epoch: 2029, cost 14.38 s 2023-09-04 10:47:08,977 44k INFO ====> Epoch: 2030, cost 14.29 s 2023-09-04 10:47:23,076 44k INFO ====> Epoch: 2031, cost 14.10 s 2023-09-04 10:47:37,093 44k INFO ====> Epoch: 2032, cost 14.02 s 2023-09-04 10:47:51,376 44k INFO ====> Epoch: 2033, cost 14.28 s 2023-09-04 10:48:05,875 44k INFO ====> Epoch: 2034, cost 14.50 s 2023-09-04 10:48:20,058 44k INFO ====> Epoch: 2035, cost 14.18 s 2023-09-04 10:48:34,231 44k INFO ====> Epoch: 2036, cost 14.17 s 2023-09-04 10:48:48,490 44k INFO ====> Epoch: 2037, cost 14.26 s 2023-09-04 10:49:02,849 44k INFO ====> Epoch: 2038, cost 14.36 s 2023-09-04 10:49:04,300 44k INFO Train Epoch: 2039 [5%] 2023-09-04 10:49:04,301 44k INFO Losses: [2.3664097785949707, 2.389932632446289, 6.941583156585693, 15.470067977905273, 0.5828575491905212], step: 42800, lr: 7.750979091648653e-05, reference_loss: 27.750850677490234 2023-09-04 10:49:17,287 44k INFO ====> Epoch: 2039, cost 14.44 s 2023-09-04 10:49:31,227 44k INFO ====> Epoch: 2040, cost 13.94 s 2023-09-04 10:49:45,303 44k INFO ====> Epoch: 2041, cost 14.08 s 2023-09-04 10:49:59,602 44k INFO ====> Epoch: 2042, cost 14.30 s 2023-09-04 10:50:13,647 44k INFO ====> Epoch: 2043, cost 14.04 s 2023-09-04 10:50:27,762 44k INFO ====> Epoch: 2044, cost 14.11 s 2023-09-04 10:50:41,803 44k INFO ====> Epoch: 2045, cost 14.04 s 2023-09-04 10:50:56,182 44k INFO ====> Epoch: 2046, cost 14.38 s 2023-09-04 10:51:10,442 44k INFO ====> Epoch: 2047, cost 14.26 s 2023-09-04 10:51:19,251 44k INFO Train Epoch: 2048 [57%] 2023-09-04 10:51:19,251 44k INFO Losses: [2.402899742126465, 2.383615255355835, 6.410867691040039, 16.17316246032715, 0.6047775149345398], step: 43000, lr: 7.742263598824878e-05, reference_loss: 27.975322723388672 2023-09-04 10:51:24,658 44k INFO ====> Epoch: 2048, cost 14.22 s 2023-09-04 10:51:38,653 44k INFO ====> Epoch: 2049, cost 13.99 s 2023-09-04 10:51:52,818 44k INFO ====> Epoch: 2050, cost 14.16 s 2023-09-04 10:52:07,097 44k INFO ====> Epoch: 2051, cost 14.28 s 2023-09-04 10:52:21,042 44k INFO ====> Epoch: 2052, cost 13.94 s 2023-09-04 10:52:35,062 44k INFO ====> Epoch: 2053, cost 14.02 s 2023-09-04 10:52:49,215 44k INFO ====> Epoch: 2054, cost 14.15 s 2023-09-04 10:53:03,421 44k INFO ====> Epoch: 2055, cost 14.21 s 2023-09-04 10:53:17,653 44k INFO ====> Epoch: 2056, cost 14.23 s 2023-09-04 10:53:31,655 44k INFO ====> Epoch: 2057, cost 14.00 s 2023-09-04 10:53:33,677 44k INFO Train Epoch: 2058 [10%] 2023-09-04 10:53:33,677 44k INFO Losses: [2.475325584411621, 2.400082588195801, 7.496735572814941, 15.040778160095215, 0.7673426270484924], step: 43200, lr: 7.732591211291238e-05, reference_loss: 28.180265426635742 2023-09-04 10:53:38,352 44k INFO Saving model and optimizer state at iteration 2058 to ./logs\44k\G_43200.pth 2023-09-04 10:53:38,938 44k INFO Saving model and optimizer state at iteration 2058 to ./logs\44k\D_43200.pth 2023-09-04 10:53:40,042 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_40800.pth 2023-09-04 10:53:40,076 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_40800.pth 2023-09-04 10:53:52,454 44k INFO ====> Epoch: 2058, cost 20.80 s 2023-09-04 10:54:06,905 44k INFO ====> Epoch: 2059, cost 14.45 s 2023-09-04 10:54:21,119 44k INFO ====> Epoch: 2060, cost 14.21 s 2023-09-04 10:54:35,169 44k INFO ====> Epoch: 2061, cost 14.05 s 2023-09-04 10:54:49,528 44k INFO ====> Epoch: 2062, cost 14.36 s 2023-09-04 10:55:03,772 44k INFO ====> Epoch: 2063, cost 14.24 s 2023-09-04 10:55:17,803 44k INFO ====> Epoch: 2064, cost 14.03 s 2023-09-04 10:55:31,883 44k INFO ====> Epoch: 2065, cost 14.08 s 2023-09-04 10:55:46,008 44k INFO ====> Epoch: 2066, cost 14.12 s 2023-09-04 10:55:55,795 44k INFO Train Epoch: 2067 [62%] 2023-09-04 10:55:55,795 44k INFO Losses: [2.2415225505828857, 2.663398265838623, 6.7893781661987305, 16.323213577270508, 0.5168402791023254], step: 43400, lr: 7.723896394492697e-05, reference_loss: 28.534353256225586 2023-09-04 10:56:00,689 44k INFO ====> Epoch: 2067, cost 14.68 s 2023-09-04 10:56:15,019 44k INFO ====> Epoch: 2068, cost 14.33 s 2023-09-04 10:56:29,083 44k INFO ====> Epoch: 2069, cost 14.06 s 2023-09-04 10:56:43,263 44k INFO ====> Epoch: 2070, cost 14.18 s 2023-09-04 10:56:57,320 44k INFO ====> Epoch: 2071, cost 14.06 s 2023-09-04 10:57:11,469 44k INFO ====> Epoch: 2072, cost 14.15 s 2023-09-04 10:57:25,488 44k INFO ====> Epoch: 2073, cost 14.02 s 2023-09-04 10:57:39,537 44k INFO ====> Epoch: 2074, cost 14.05 s 2023-09-04 10:57:53,709 44k INFO ====> Epoch: 2075, cost 14.17 s 2023-09-04 10:58:07,815 44k INFO ====> Epoch: 2076, cost 14.11 s 2023-09-04 10:58:10,569 44k INFO Train Epoch: 2077 [14%] 2023-09-04 10:58:10,569 44k INFO Losses: [2.265763282775879, 2.4624826908111572, 8.303570747375488, 17.314847946166992, 0.6018045544624329], step: 43600, lr: 7.714246953054337e-05, reference_loss: 30.948469161987305 2023-09-04 10:58:22,177 44k INFO ====> Epoch: 2077, cost 14.36 s 2023-09-04 10:58:36,378 44k INFO ====> Epoch: 2078, cost 14.20 s 2023-09-04 10:58:50,650 44k INFO ====> Epoch: 2079, cost 14.27 s 2023-09-04 10:59:04,933 44k INFO ====> Epoch: 2080, cost 14.28 s 2023-09-04 10:59:19,046 44k INFO ====> Epoch: 2081, cost 14.11 s 2023-09-04 10:59:32,928 44k INFO ====> Epoch: 2082, cost 13.88 s 2023-09-04 10:59:47,165 44k INFO ====> Epoch: 2083, cost 14.24 s 2023-09-04 11:00:01,475 44k INFO ====> Epoch: 2084, cost 14.31 s 2023-09-04 11:00:15,872 44k INFO ====> Epoch: 2085, cost 14.40 s 2023-09-04 11:00:26,100 44k INFO Train Epoch: 2086 [67%] 2023-09-04 11:00:26,101 44k INFO Losses: [2.3662543296813965, 2.4371395111083984, 6.044337749481201, 13.606073379516602, 0.5626529455184937], step: 43800, lr: 7.705572763230678e-05, reference_loss: 25.016456604003906 2023-09-04 11:00:30,088 44k INFO ====> Epoch: 2086, cost 14.22 s 2023-09-04 11:00:44,358 44k INFO ====> Epoch: 2087, cost 14.27 s 2023-09-04 11:00:58,629 44k INFO ====> Epoch: 2088, cost 14.27 s 2023-09-04 11:01:12,852 44k INFO ====> Epoch: 2089, cost 14.22 s 2023-09-04 11:01:26,841 44k INFO ====> Epoch: 2090, cost 13.99 s 2023-09-04 11:01:40,950 44k INFO ====> Epoch: 2091, cost 14.11 s 2023-09-04 11:01:55,114 44k INFO ====> Epoch: 2092, cost 14.16 s 2023-09-04 11:02:09,413 44k INFO ====> Epoch: 2093, cost 14.30 s 2023-09-04 11:02:23,643 44k INFO ====> Epoch: 2094, cost 14.23 s 2023-09-04 11:02:37,801 44k INFO ====> Epoch: 2095, cost 14.16 s 2023-09-04 11:02:41,275 44k INFO Train Epoch: 2096 [19%] 2023-09-04 11:02:41,275 44k INFO Losses: [2.2415518760681152, 2.934481620788574, 8.379334449768066, 16.093034744262695, 0.7513843774795532], step: 44000, lr: 7.695946213451885e-05, reference_loss: 30.39978790283203 2023-09-04 11:02:45,945 44k INFO Saving model and optimizer state at iteration 2096 to ./logs\44k\G_44000.pth 2023-09-04 11:02:46,535 44k INFO Saving model and optimizer state at iteration 2096 to ./logs\44k\D_44000.pth 2023-09-04 11:02:47,757 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_41600.pth 2023-09-04 11:02:47,789 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_41600.pth 2023-09-04 11:02:58,585 44k INFO ====> Epoch: 2096, cost 20.78 s 2023-09-04 11:03:12,761 44k INFO ====> Epoch: 2097, cost 14.18 s 2023-09-04 11:03:26,695 44k INFO ====> Epoch: 2098, cost 13.93 s 2023-09-04 11:03:40,695 44k INFO ====> Epoch: 2099, cost 14.00 s 2023-09-04 11:03:54,864 44k INFO ====> Epoch: 2100, cost 14.17 s 2023-09-04 11:04:09,084 44k INFO ====> Epoch: 2101, cost 14.22 s 2023-09-04 11:04:23,381 44k INFO ====> Epoch: 2102, cost 14.30 s 2023-09-04 11:04:37,579 44k INFO ====> Epoch: 2103, cost 14.20 s 2023-09-04 11:04:51,651 44k INFO ====> Epoch: 2104, cost 14.07 s 2023-09-04 11:05:02,707 44k INFO Train Epoch: 2105 [71%] 2023-09-04 11:05:02,708 44k INFO Losses: [2.4753448963165283, 2.2937874794006348, 5.999752044677734, 14.32504653930664, 0.7621451616287231], step: 44200, lr: 7.687292601669116e-05, reference_loss: 25.856077194213867 2023-09-04 11:05:06,166 44k INFO ====> Epoch: 2105, cost 14.51 s 2023-09-04 11:05:20,229 44k INFO ====> Epoch: 2106, cost 14.06 s 2023-09-04 11:05:34,320 44k INFO ====> Epoch: 2107, cost 14.09 s 2023-09-04 11:05:48,461 44k INFO ====> Epoch: 2108, cost 14.14 s 2023-09-04 11:06:02,875 44k INFO ====> Epoch: 2109, cost 14.41 s 2023-09-04 11:06:17,014 44k INFO ====> Epoch: 2110, cost 14.14 s 2023-09-04 11:06:31,240 44k INFO ====> Epoch: 2111, cost 14.23 s 2023-09-04 11:06:45,568 44k INFO ====> Epoch: 2112, cost 14.33 s 2023-09-04 11:06:59,946 44k INFO ====> Epoch: 2113, cost 14.38 s 2023-09-04 11:07:14,151 44k INFO ====> Epoch: 2114, cost 14.20 s 2023-09-04 11:07:18,207 44k INFO Train Epoch: 2115 [24%] 2023-09-04 11:07:18,207 44k INFO Losses: [2.344189167022705, 2.3280892372131348, 7.185760974884033, 16.178421020507812, 0.5273241400718689], step: 44400, lr: 7.677688889243324e-05, reference_loss: 28.563785552978516 2023-09-04 11:07:28,327 44k INFO ====> Epoch: 2115, cost 14.18 s 2023-09-04 11:07:42,400 44k INFO ====> Epoch: 2116, cost 14.07 s 2023-09-04 11:07:56,578 44k INFO ====> Epoch: 2117, cost 14.18 s 2023-09-04 11:08:10,698 44k INFO ====> Epoch: 2118, cost 14.12 s 2023-09-04 11:08:24,876 44k INFO ====> Epoch: 2119, cost 14.18 s 2023-09-04 11:08:39,094 44k INFO ====> Epoch: 2120, cost 14.22 s 2023-09-04 11:08:53,365 44k INFO ====> Epoch: 2121, cost 14.27 s 2023-09-04 11:09:07,623 44k INFO ====> Epoch: 2122, cost 14.26 s 2023-09-04 11:09:21,623 44k INFO ====> Epoch: 2123, cost 14.00 s 2023-09-04 11:09:33,151 44k INFO Train Epoch: 2124 [76%] 2023-09-04 11:09:33,151 44k INFO Losses: [2.172776699066162, 2.8807716369628906, 7.604310035705566, 13.850197792053223, 0.5019271969795227], step: 44600, lr: 7.669055806683539e-05, reference_loss: 27.00998306274414 2023-09-04 11:09:35,849 44k INFO ====> Epoch: 2124, cost 14.23 s 2023-09-04 11:09:49,860 44k INFO ====> Epoch: 2125, cost 14.01 s 2023-09-04 11:10:04,105 44k INFO ====> Epoch: 2126, cost 14.24 s 2023-09-04 11:10:18,097 44k INFO ====> Epoch: 2127, cost 13.99 s 2023-09-04 11:10:32,127 44k INFO ====> Epoch: 2128, cost 14.03 s 2023-09-04 11:10:46,157 44k INFO ====> Epoch: 2129, cost 14.03 s 2023-09-04 11:11:00,419 44k INFO ====> Epoch: 2130, cost 14.26 s 2023-09-04 11:11:14,681 44k INFO ====> Epoch: 2131, cost 14.26 s 2023-09-04 11:11:28,776 44k INFO ====> Epoch: 2132, cost 14.10 s 2023-09-04 11:11:42,797 44k INFO ====> Epoch: 2133, cost 14.02 s 2023-09-04 11:11:47,653 44k INFO Train Epoch: 2134 [29%] 2023-09-04 11:11:47,653 44k INFO Losses: [2.3996944427490234, 2.35332989692688, 9.194993019104004, 18.135066986083984, 0.7650825381278992], step: 44800, lr: 7.659474877433003e-05, reference_loss: 32.848167419433594 2023-09-04 11:11:52,318 44k INFO Saving model and optimizer state at iteration 2134 to ./logs\44k\G_44800.pth 2023-09-04 11:11:53,002 44k INFO Saving model and optimizer state at iteration 2134 to ./logs\44k\D_44800.pth 2023-09-04 11:11:54,167 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_42400.pth 2023-09-04 11:11:54,201 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_42400.pth 2023-09-04 11:12:03,670 44k INFO ====> Epoch: 2134, cost 20.87 s 2023-09-04 11:12:18,050 44k INFO ====> Epoch: 2135, cost 14.38 s 2023-09-04 11:12:32,098 44k INFO ====> Epoch: 2136, cost 14.05 s 2023-09-04 11:12:46,367 44k INFO ====> Epoch: 2137, cost 14.27 s 2023-09-04 11:13:00,713 44k INFO ====> Epoch: 2138, cost 14.35 s 2023-09-04 11:13:14,969 44k INFO ====> Epoch: 2139, cost 14.26 s 2023-09-04 11:13:29,033 44k INFO ====> Epoch: 2140, cost 14.06 s 2023-09-04 11:13:43,046 44k INFO ====> Epoch: 2141, cost 14.01 s 2023-09-04 11:13:57,191 44k INFO ====> Epoch: 2142, cost 14.14 s 2023-09-04 11:14:09,545 44k INFO Train Epoch: 2143 [81%] 2023-09-04 11:14:09,546 44k INFO Losses: [2.238461971282959, 2.564422130584717, 8.549482345581055, 17.234840393066406, 0.7005695700645447], step: 45000, lr: 7.65086227539411e-05, reference_loss: 31.287776947021484 2023-09-04 11:14:11,582 44k INFO ====> Epoch: 2143, cost 14.39 s 2023-09-04 11:14:25,739 44k INFO ====> Epoch: 2144, cost 14.16 s 2023-09-04 11:14:39,699 44k INFO ====> Epoch: 2145, cost 13.96 s 2023-09-04 11:14:53,773 44k INFO ====> Epoch: 2146, cost 14.07 s 2023-09-04 11:15:07,911 44k INFO ====> Epoch: 2147, cost 14.14 s 2023-09-04 11:15:21,904 44k INFO ====> Epoch: 2148, cost 13.99 s 2023-09-04 11:15:35,797 44k INFO ====> Epoch: 2149, cost 13.89 s 2023-09-04 11:15:49,888 44k INFO ====> Epoch: 2150, cost 14.09 s 2023-09-04 11:16:04,124 44k INFO ====> Epoch: 2151, cost 14.24 s 2023-09-04 11:16:18,336 44k INFO ====> Epoch: 2152, cost 14.21 s 2023-09-04 11:16:23,721 44k INFO Train Epoch: 2153 [33%] 2023-09-04 11:16:23,721 44k INFO Losses: [2.495920419692993, 2.5033411979675293, 9.961597442626953, 18.173418045043945, 0.6607897877693176], step: 45200, lr: 7.641304075269622e-05, reference_loss: 33.795066833496094 2023-09-04 11:16:32,580 44k INFO ====> Epoch: 2153, cost 14.24 s 2023-09-04 11:16:46,868 44k INFO ====> Epoch: 2154, cost 14.29 s 2023-09-04 11:17:01,238 44k INFO ====> Epoch: 2155, cost 14.37 s 2023-09-04 11:17:15,519 44k INFO ====> Epoch: 2156, cost 14.28 s 2023-09-04 11:17:29,577 44k INFO ====> Epoch: 2157, cost 14.06 s 2023-09-04 11:17:43,723 44k INFO ====> Epoch: 2158, cost 14.15 s 2023-09-04 11:17:58,239 44k INFO ====> Epoch: 2159, cost 14.52 s 2023-09-04 11:18:12,570 44k INFO ====> Epoch: 2160, cost 14.33 s 2023-09-04 11:18:26,661 44k INFO ====> Epoch: 2161, cost 14.09 s 2023-09-04 11:18:39,631 44k INFO Train Epoch: 2162 [86%] 2023-09-04 11:18:39,631 44k INFO Losses: [2.494779586791992, 2.438199043273926, 6.210604190826416, 15.009268760681152, 0.7156621813774109], step: 45400, lr: 7.632711905165067e-05, reference_loss: 26.868513107299805 2023-09-04 11:18:40,972 44k INFO ====> Epoch: 2162, cost 14.31 s 2023-09-04 11:18:55,438 44k INFO ====> Epoch: 2163, cost 14.47 s 2023-09-04 11:19:09,759 44k INFO ====> Epoch: 2164, cost 14.32 s 2023-09-04 11:19:23,624 44k INFO ====> Epoch: 2165, cost 13.87 s 2023-09-04 11:19:37,543 44k INFO ====> Epoch: 2166, cost 13.92 s 2023-09-04 11:19:51,960 44k INFO ====> Epoch: 2167, cost 14.42 s 2023-09-04 11:20:06,029 44k INFO ====> Epoch: 2168, cost 14.07 s 2023-09-04 11:20:20,233 44k INFO ====> Epoch: 2169, cost 14.20 s 2023-09-04 11:20:34,341 44k INFO ====> Epoch: 2170, cost 14.11 s 2023-09-04 11:20:48,518 44k INFO ====> Epoch: 2171, cost 14.18 s 2023-09-04 11:20:54,663 44k INFO Train Epoch: 2172 [38%] 2023-09-04 11:20:54,663 44k INFO Losses: [2.6345772743225098, 2.152156352996826, 6.5477190017700195, 15.450660705566406, 0.6512016654014587], step: 45600, lr: 7.623176380245642e-05, reference_loss: 27.436315536499023 2023-09-04 11:20:59,434 44k INFO Saving model and optimizer state at iteration 2172 to ./logs\44k\G_45600.pth 2023-09-04 11:21:00,051 44k INFO Saving model and optimizer state at iteration 2172 to ./logs\44k\D_45600.pth 2023-09-04 11:21:01,628 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_43200.pth 2023-09-04 11:21:01,661 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_43200.pth 2023-09-04 11:21:09,716 44k INFO ====> Epoch: 2172, cost 21.20 s 2023-09-04 11:21:23,725 44k INFO ====> Epoch: 2173, cost 14.01 s 2023-09-04 11:21:37,574 44k INFO ====> Epoch: 2174, cost 13.85 s 2023-09-04 11:21:51,477 44k INFO ====> Epoch: 2175, cost 13.90 s 2023-09-04 11:22:05,688 44k INFO ====> Epoch: 2176, cost 14.21 s 2023-09-04 11:22:19,811 44k INFO ====> Epoch: 2177, cost 14.12 s 2023-09-04 11:22:33,826 44k INFO ====> Epoch: 2178, cost 14.01 s 2023-09-04 11:22:47,923 44k INFO ====> Epoch: 2179, cost 14.10 s 2023-09-04 11:23:02,137 44k INFO ====> Epoch: 2180, cost 14.21 s 2023-09-04 11:23:15,896 44k INFO Train Epoch: 2181 [90%] 2023-09-04 11:23:15,896 44k INFO Losses: [2.1453664302825928, 2.8553621768951416, 8.452786445617676, 17.434743881225586, 0.8859872817993164], step: 45800, lr: 7.614604593604133e-05, reference_loss: 31.774246215820312 2023-09-04 11:23:16,625 44k INFO ====> Epoch: 2181, cost 14.49 s 2023-09-04 11:23:30,607 44k INFO ====> Epoch: 2182, cost 13.98 s 2023-09-04 11:23:44,760 44k INFO ====> Epoch: 2183, cost 14.15 s 2023-09-04 11:23:59,121 44k INFO ====> Epoch: 2184, cost 14.36 s 2023-09-04 11:24:13,289 44k INFO ====> Epoch: 2185, cost 14.17 s 2023-09-04 11:24:27,457 44k INFO ====> Epoch: 2186, cost 14.17 s 2023-09-04 11:24:41,832 44k INFO ====> Epoch: 2187, cost 14.38 s 2023-09-04 11:24:56,181 44k INFO ====> Epoch: 2188, cost 14.35 s 2023-09-04 11:25:10,447 44k INFO ====> Epoch: 2189, cost 14.27 s 2023-09-04 11:25:24,281 44k INFO ====> Epoch: 2190, cost 13.83 s 2023-09-04 11:25:31,112 44k INFO Train Epoch: 2191 [43%] 2023-09-04 11:25:31,112 44k INFO Losses: [2.355067729949951, 2.557523727416992, 6.9383158683776855, 16.308889389038086, 0.5820325016975403], step: 46000, lr: 7.605091690096695e-05, reference_loss: 28.741830825805664 2023-09-04 11:25:38,536 44k INFO ====> Epoch: 2191, cost 14.25 s 2023-09-04 11:25:52,890 44k INFO ====> Epoch: 2192, cost 14.35 s 2023-09-04 11:26:07,054 44k INFO ====> Epoch: 2193, cost 14.16 s 2023-09-04 11:26:21,225 44k INFO ====> Epoch: 2194, cost 14.17 s 2023-09-04 11:26:35,291 44k INFO ====> Epoch: 2195, cost 14.07 s 2023-09-04 11:26:49,415 44k INFO ====> Epoch: 2196, cost 14.12 s 2023-09-04 11:27:03,721 44k INFO ====> Epoch: 2197, cost 14.31 s 2023-09-04 11:27:17,811 44k INFO ====> Epoch: 2198, cost 14.09 s 2023-09-04 11:27:31,884 44k INFO ====> Epoch: 2199, cost 14.07 s 2023-09-04 11:27:46,014 44k INFO Train Epoch: 2200 [95%] 2023-09-04 11:27:46,015 44k INFO Losses: [2.352694272994995, 2.514422655105591, 5.926360130310059, 14.375439643859863, 0.5249522924423218], step: 46200, lr: 7.596540238561933e-05, reference_loss: 25.69386863708496 2023-09-04 11:27:46,269 44k INFO ====> Epoch: 2200, cost 14.39 s 2023-09-04 11:28:00,563 44k INFO ====> Epoch: 2201, cost 14.29 s 2023-09-04 11:28:14,715 44k INFO ====> Epoch: 2202, cost 14.15 s 2023-09-04 11:28:28,801 44k INFO ====> Epoch: 2203, cost 14.09 s 2023-09-04 11:28:42,957 44k INFO ====> Epoch: 2204, cost 14.16 s 2023-09-04 11:28:57,136 44k INFO ====> Epoch: 2205, cost 14.18 s 2023-09-04 11:29:11,350 44k INFO ====> Epoch: 2206, cost 14.21 s 2023-09-04 11:29:25,556 44k INFO ====> Epoch: 2207, cost 14.21 s 2023-09-04 11:29:39,548 44k INFO ====> Epoch: 2208, cost 13.99 s 2023-09-04 11:29:53,925 44k INFO ====> Epoch: 2209, cost 14.38 s 2023-09-04 11:30:01,604 44k INFO Train Epoch: 2210 [48%] 2023-09-04 11:30:01,605 44k INFO Losses: [2.2976551055908203, 2.4633049964904785, 8.386139869689941, 17.746442794799805, 0.6892486214637756], step: 46400, lr: 7.587049902801034e-05, reference_loss: 31.582792282104492 2023-09-04 11:30:06,307 44k INFO Saving model and optimizer state at iteration 2210 to ./logs\44k\G_46400.pth 2023-09-04 11:30:06,941 44k INFO Saving model and optimizer state at iteration 2210 to ./logs\44k\D_46400.pth 2023-09-04 11:30:08,180 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_44000.pth 2023-09-04 11:30:08,215 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_44000.pth 2023-09-04 11:30:14,801 44k INFO ====> Epoch: 2210, cost 20.88 s 2023-09-04 11:30:28,887 44k INFO ====> Epoch: 2211, cost 14.09 s 2023-09-04 11:30:42,902 44k INFO ====> Epoch: 2212, cost 14.01 s 2023-09-04 11:30:57,408 44k INFO ====> Epoch: 2213, cost 14.51 s 2023-09-04 11:31:11,803 44k INFO ====> Epoch: 2214, cost 14.39 s 2023-09-04 11:31:25,807 44k INFO ====> Epoch: 2215, cost 14.00 s 2023-09-04 11:31:39,765 44k INFO ====> Epoch: 2216, cost 13.96 s 2023-09-04 11:31:54,214 44k INFO ====> Epoch: 2217, cost 14.45 s 2023-09-04 11:32:08,262 44k INFO ====> Epoch: 2218, cost 14.05 s 2023-09-04 11:32:22,281 44k INFO ====> Epoch: 2219, cost 14.02 s 2023-09-04 11:32:22,961 44k INFO Train Epoch: 2220 [0%] 2023-09-04 11:32:22,961 44k INFO Losses: [2.424175262451172, 2.851088047027588, 7.267372131347656, 16.636659622192383, 0.6497906446456909], step: 46600, lr: 7.577571423289166e-05, reference_loss: 29.829084396362305 2023-09-04 11:32:36,637 44k INFO ====> Epoch: 2220, cost 14.36 s 2023-09-04 11:32:50,926 44k INFO ====> Epoch: 2221, cost 14.29 s 2023-09-04 11:33:05,073 44k INFO ====> Epoch: 2222, cost 14.15 s 2023-09-04 11:33:19,130 44k INFO ====> Epoch: 2223, cost 14.06 s 2023-09-04 11:33:33,182 44k INFO ====> Epoch: 2224, cost 14.05 s 2023-09-04 11:33:47,190 44k INFO ====> Epoch: 2225, cost 14.01 s 2023-09-04 11:34:01,297 44k INFO ====> Epoch: 2226, cost 14.11 s 2023-09-04 11:34:15,373 44k INFO ====> Epoch: 2227, cost 14.08 s 2023-09-04 11:34:29,383 44k INFO ====> Epoch: 2228, cost 14.01 s 2023-09-04 11:34:37,524 44k INFO Train Epoch: 2229 [52%] 2023-09-04 11:34:37,524 44k INFO Losses: [2.2416913509368896, 2.7737436294555664, 7.856834888458252, 17.17852020263672, 0.7312353253364563], step: 46800, lr: 7.569050916578924e-05, reference_loss: 30.782026290893555 2023-09-04 11:34:43,643 44k INFO ====> Epoch: 2229, cost 14.26 s 2023-09-04 11:34:57,869 44k INFO ====> Epoch: 2230, cost 14.23 s 2023-09-04 11:35:11,966 44k INFO ====> Epoch: 2231, cost 14.10 s 2023-09-04 11:35:26,017 44k INFO ====> Epoch: 2232, cost 14.05 s 2023-09-04 11:35:39,910 44k INFO ====> Epoch: 2233, cost 13.89 s 2023-09-04 11:35:54,411 44k INFO ====> Epoch: 2234, cost 14.50 s 2023-09-04 11:36:08,833 44k INFO ====> Epoch: 2235, cost 14.42 s 2023-09-04 11:36:22,933 44k INFO ====> Epoch: 2236, cost 14.10 s 2023-09-04 11:36:36,990 44k INFO ====> Epoch: 2237, cost 14.06 s 2023-09-04 11:36:51,337 44k INFO ====> Epoch: 2238, cost 14.35 s 2023-09-04 11:36:52,761 44k INFO Train Epoch: 2239 [5%] 2023-09-04 11:36:52,762 44k INFO Losses: [2.2508931159973145, 2.6784913539886475, 7.849937915802002, 17.198352813720703, 0.5709038376808167], step: 47000, lr: 7.559594923148515e-05, reference_loss: 30.5485782623291 2023-09-04 11:37:05,881 44k INFO ====> Epoch: 2239, cost 14.54 s 2023-09-04 11:37:19,929 44k INFO ====> Epoch: 2240, cost 14.05 s 2023-09-04 11:37:33,902 44k INFO ====> Epoch: 2241, cost 13.97 s 2023-09-04 11:37:48,093 44k INFO ====> Epoch: 2242, cost 14.19 s 2023-09-04 11:38:02,592 44k INFO ====> Epoch: 2243, cost 14.50 s 2023-09-04 11:38:16,860 44k INFO ====> Epoch: 2244, cost 14.27 s 2023-09-04 11:38:30,836 44k INFO ====> Epoch: 2245, cost 13.98 s 2023-09-04 11:38:44,994 44k INFO ====> Epoch: 2246, cost 14.16 s 2023-09-04 11:38:59,060 44k INFO ====> Epoch: 2247, cost 14.07 s 2023-09-04 11:39:08,089 44k INFO Train Epoch: 2248 [57%] 2023-09-04 11:39:08,090 44k INFO Losses: [2.476889133453369, 2.2090048789978027, 6.605546951293945, 15.534485816955566, 0.7860794067382812], step: 47200, lr: 7.5510946298921e-05, reference_loss: 27.61200714111328 2023-09-04 11:39:12,811 44k INFO Saving model and optimizer state at iteration 2248 to ./logs\44k\G_47200.pth 2023-09-04 11:39:13,415 44k INFO Saving model and optimizer state at iteration 2248 to ./logs\44k\D_47200.pth 2023-09-04 11:39:14,572 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_44800.pth 2023-09-04 11:39:14,608 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_44800.pth 2023-09-04 11:39:19,822 44k INFO ====> Epoch: 2248, cost 20.76 s 2023-09-04 11:39:33,910 44k INFO ====> Epoch: 2249, cost 14.09 s 2023-09-04 11:39:47,996 44k INFO ====> Epoch: 2250, cost 14.09 s 2023-09-04 11:40:02,169 44k INFO ====> Epoch: 2251, cost 14.17 s 2023-09-04 11:40:16,344 44k INFO ====> Epoch: 2252, cost 14.17 s 2023-09-04 11:40:30,286 44k INFO ====> Epoch: 2253, cost 13.94 s 2023-09-04 11:40:44,484 44k INFO ====> Epoch: 2254, cost 14.20 s 2023-09-04 11:40:58,708 44k INFO ====> Epoch: 2255, cost 14.22 s 2023-09-04 11:41:12,812 44k INFO ====> Epoch: 2256, cost 14.10 s 2023-09-04 11:41:26,729 44k INFO ====> Epoch: 2257, cost 13.92 s 2023-09-04 11:41:28,740 44k INFO Train Epoch: 2258 [10%] 2023-09-04 11:41:28,740 44k INFO Losses: [2.6248116493225098, 2.1823298931121826, 6.407721042633057, 15.030617713928223, 0.7183821797370911], step: 47400, lr: 7.541661069198744e-05, reference_loss: 26.9638614654541 2023-09-04 11:41:41,029 44k INFO ====> Epoch: 2258, cost 14.30 s 2023-09-04 11:41:55,483 44k INFO ====> Epoch: 2259, cost 14.45 s 2023-09-04 11:42:09,896 44k INFO ====> Epoch: 2260, cost 14.41 s 2023-09-04 11:42:24,097 44k INFO ====> Epoch: 2261, cost 14.20 s 2023-09-04 11:42:38,285 44k INFO ====> Epoch: 2262, cost 14.19 s 2023-09-04 11:42:52,656 44k INFO ====> Epoch: 2263, cost 14.37 s 2023-09-04 11:43:07,043 44k INFO ====> Epoch: 2264, cost 14.39 s 2023-09-04 11:43:20,982 44k INFO ====> Epoch: 2265, cost 13.94 s 2023-09-04 11:43:34,940 44k INFO ====> Epoch: 2266, cost 13.96 s 2023-09-04 11:43:44,543 44k INFO Train Epoch: 2267 [62%] 2023-09-04 11:43:44,543 44k INFO Losses: [2.414346218109131, 2.5064947605133057, 7.265851020812988, 17.149412155151367, 0.6273826956748962], step: 47600, lr: 7.533180941443172e-05, reference_loss: 29.96348762512207 2023-09-04 11:43:49,354 44k INFO ====> Epoch: 2267, cost 14.41 s 2023-09-04 11:44:03,731 44k INFO ====> Epoch: 2268, cost 14.38 s 2023-09-04 11:44:17,950 44k INFO ====> Epoch: 2269, cost 14.22 s 2023-09-04 11:44:32,141 44k INFO ====> Epoch: 2270, cost 14.19 s 2023-09-04 11:44:46,260 44k INFO ====> Epoch: 2271, cost 14.12 s 2023-09-04 11:45:00,442 44k INFO ====> Epoch: 2272, cost 14.18 s 2023-09-04 11:45:14,605 44k INFO ====> Epoch: 2273, cost 14.16 s 2023-09-04 11:45:28,599 44k INFO ====> Epoch: 2274, cost 13.99 s 2023-09-04 11:45:42,675 44k INFO ====> Epoch: 2275, cost 14.08 s 2023-09-04 11:45:56,949 44k INFO ====> Epoch: 2276, cost 14.27 s 2023-09-04 11:45:59,712 44k INFO Train Epoch: 2277 [14%] 2023-09-04 11:45:59,712 44k INFO Losses: [2.428086519241333, 2.3139705657958984, 7.569002151489258, 16.991985321044922, 0.617313027381897], step: 47800, lr: 7.523769760269012e-05, reference_loss: 29.92035675048828 2023-09-04 11:46:11,258 44k INFO ====> Epoch: 2277, cost 14.31 s 2023-09-04 11:46:25,381 44k INFO ====> Epoch: 2278, cost 14.12 s 2023-09-04 11:46:39,284 44k INFO ====> Epoch: 2279, cost 13.90 s 2023-09-04 11:46:53,451 44k INFO ====> Epoch: 2280, cost 14.17 s 2023-09-04 11:47:07,636 44k INFO ====> Epoch: 2281, cost 14.19 s 2023-09-04 11:47:21,765 44k INFO ====> Epoch: 2282, cost 14.13 s 2023-09-04 11:47:35,785 44k INFO ====> Epoch: 2283, cost 14.02 s 2023-09-04 11:47:50,100 44k INFO ====> Epoch: 2284, cost 14.32 s 2023-09-04 11:48:04,501 44k INFO ====> Epoch: 2285, cost 14.40 s 2023-09-04 11:48:14,895 44k INFO Train Epoch: 2286 [67%] 2023-09-04 11:48:14,895 44k INFO Losses: [2.4973809719085693, 2.336699962615967, 7.499048709869385, 17.23822021484375, 0.6476346850395203], step: 48000, lr: 7.51530975017506e-05, reference_loss: 30.218984603881836 2023-09-04 11:48:19,552 44k INFO Saving model and optimizer state at iteration 2286 to ./logs\44k\G_48000.pth 2023-09-04 11:48:20,286 44k INFO Saving model and optimizer state at iteration 2286 to ./logs\44k\D_48000.pth 2023-09-04 11:48:21,423 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_45600.pth 2023-09-04 11:48:21,458 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_45600.pth 2023-09-04 11:48:25,274 44k INFO ====> Epoch: 2286, cost 20.77 s 2023-09-04 11:48:39,361 44k INFO ====> Epoch: 2287, cost 14.09 s 2023-09-04 11:48:53,864 44k INFO ====> Epoch: 2288, cost 14.50 s 2023-09-04 11:49:08,278 44k INFO ====> Epoch: 2289, cost 14.41 s 2023-09-04 11:49:22,221 44k INFO ====> Epoch: 2290, cost 13.94 s 2023-09-04 11:49:36,185 44k INFO ====> Epoch: 2291, cost 13.96 s 2023-09-04 11:49:50,445 44k INFO ====> Epoch: 2292, cost 14.26 s 2023-09-04 11:50:04,764 44k INFO ====> Epoch: 2293, cost 14.32 s 2023-09-04 11:50:18,962 44k INFO ====> Epoch: 2294, cost 14.20 s 2023-09-04 11:50:32,974 44k INFO ====> Epoch: 2295, cost 14.01 s 2023-09-04 11:50:36,383 44k INFO Train Epoch: 2296 [19%] 2023-09-04 11:50:36,383 44k INFO Losses: [2.523473024368286, 2.272891044616699, 7.397848129272461, 17.225889205932617, 0.6718478798866272], step: 48200, lr: 7.505920895428489e-05, reference_loss: 30.091949462890625 2023-09-04 11:50:47,449 44k INFO ====> Epoch: 2296, cost 14.48 s 2023-09-04 11:51:01,672 44k INFO ====> Epoch: 2297, cost 14.22 s 2023-09-04 11:51:15,739 44k INFO ====> Epoch: 2298, cost 14.07 s 2023-09-04 11:51:29,710 44k INFO ====> Epoch: 2299, cost 13.97 s 2023-09-04 11:51:43,616 44k INFO ====> Epoch: 2300, cost 13.91 s 2023-09-04 11:51:57,769 44k INFO ====> Epoch: 2301, cost 14.15 s 2023-09-04 11:52:11,835 44k INFO ====> Epoch: 2302, cost 14.07 s 2023-09-04 11:52:25,877 44k INFO ====> Epoch: 2303, cost 14.04 s 2023-09-04 11:52:39,912 44k INFO ====> Epoch: 2304, cost 14.04 s 2023-09-04 11:52:51,028 44k INFO Train Epoch: 2305 [71%] 2023-09-04 11:52:51,028 44k INFO Losses: [2.457149028778076, 2.394495725631714, 6.676560878753662, 15.09030532836914, 0.6388325095176697], step: 48400, lr: 7.497480955270424e-05, reference_loss: 27.257343292236328 2023-09-04 11:52:54,459 44k INFO ====> Epoch: 2305, cost 14.55 s 2023-09-04 11:53:08,696 44k INFO ====> Epoch: 2306, cost 14.24 s 2023-09-04 11:53:22,662 44k INFO ====> Epoch: 2307, cost 13.97 s 2023-09-04 11:53:36,561 44k INFO ====> Epoch: 2308, cost 13.90 s 2023-09-04 11:53:50,804 44k INFO ====> Epoch: 2309, cost 14.24 s 2023-09-04 11:54:05,089 44k INFO ====> Epoch: 2310, cost 14.29 s 2023-09-04 11:54:19,293 44k INFO ====> Epoch: 2311, cost 14.20 s 2023-09-04 11:54:33,373 44k INFO ====> Epoch: 2312, cost 14.08 s 2023-09-04 11:54:47,749 44k INFO ====> Epoch: 2313, cost 14.38 s 2023-09-04 11:55:01,954 44k INFO ====> Epoch: 2314, cost 14.20 s 2023-09-04 11:55:06,162 44k INFO Train Epoch: 2315 [24%] 2023-09-04 11:55:06,162 44k INFO Losses: [2.2868704795837402, 2.648766279220581, 7.967991828918457, 15.183881759643555, 0.7274500727653503], step: 48600, lr: 7.488114373985791e-05, reference_loss: 28.814958572387695 2023-09-04 11:55:16,463 44k INFO ====> Epoch: 2315, cost 14.51 s 2023-09-04 11:55:30,407 44k INFO ====> Epoch: 2316, cost 13.94 s 2023-09-04 11:55:44,427 44k INFO ====> Epoch: 2317, cost 14.02 s 2023-09-04 11:55:58,635 44k INFO ====> Epoch: 2318, cost 14.21 s 2023-09-04 11:56:12,825 44k INFO ====> Epoch: 2319, cost 14.19 s 2023-09-04 11:56:26,933 44k INFO ====> Epoch: 2320, cost 14.11 s 2023-09-04 11:56:41,042 44k INFO ====> Epoch: 2321, cost 14.11 s 2023-09-04 11:56:55,224 44k INFO ====> Epoch: 2322, cost 14.18 s 2023-09-04 11:57:09,555 44k INFO ====> Epoch: 2323, cost 14.33 s 2023-09-04 11:57:20,933 44k INFO Train Epoch: 2324 [76%] 2023-09-04 11:57:20,933 44k INFO Losses: [2.201587200164795, 2.487565040588379, 9.036664962768555, 17.782093048095703, 0.6736162304878235], step: 48800, lr: 7.479694456151102e-05, reference_loss: 32.18152618408203 2023-09-04 11:57:25,593 44k INFO Saving model and optimizer state at iteration 2324 to ./logs\44k\G_48800.pth 2023-09-04 11:57:26,716 44k INFO Saving model and optimizer state at iteration 2324 to ./logs\44k\D_48800.pth 2023-09-04 11:57:28,328 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_46400.pth 2023-09-04 11:57:28,360 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_46400.pth 2023-09-04 11:57:30,809 44k INFO ====> Epoch: 2324, cost 21.25 s 2023-09-04 11:57:44,776 44k INFO ====> Epoch: 2325, cost 13.97 s 2023-09-04 11:57:58,946 44k INFO ====> Epoch: 2326, cost 14.17 s 2023-09-04 11:58:13,064 44k INFO ====> Epoch: 2327, cost 14.12 s 2023-09-04 11:58:27,088 44k INFO ====> Epoch: 2328, cost 14.02 s 2023-09-04 11:58:41,092 44k INFO ====> Epoch: 2329, cost 14.01 s 2023-09-04 11:58:55,418 44k INFO ====> Epoch: 2330, cost 14.33 s 2023-09-04 11:59:09,601 44k INFO ====> Epoch: 2331, cost 14.18 s 2023-09-04 11:59:23,614 44k INFO ====> Epoch: 2332, cost 14.01 s 2023-09-04 11:59:37,618 44k INFO ====> Epoch: 2333, cost 14.00 s 2023-09-04 11:59:42,504 44k INFO Train Epoch: 2334 [29%] 2023-09-04 11:59:42,504 44k INFO Losses: [2.1804208755493164, 2.9398000240325928, 6.624731540679932, 15.489228248596191, 0.6505204439163208], step: 49000, lr: 7.470350095488404e-05, reference_loss: 27.884700775146484 2023-09-04 11:59:52,282 44k INFO ====> Epoch: 2334, cost 14.67 s 2023-09-04 12:00:06,705 44k INFO ====> Epoch: 2335, cost 14.42 s 2023-09-04 12:00:20,921 44k INFO ====> Epoch: 2336, cost 14.22 s 2023-09-04 12:00:35,028 44k INFO ====> Epoch: 2337, cost 14.11 s 2023-09-04 12:00:49,367 44k INFO ====> Epoch: 2338, cost 14.34 s 2023-09-04 12:01:03,725 44k INFO ====> Epoch: 2339, cost 14.36 s 2023-09-04 12:01:17,847 44k INFO ====> Epoch: 2340, cost 14.12 s 2023-09-04 12:01:31,861 44k INFO ====> Epoch: 2341, cost 14.01 s 2023-09-04 12:01:45,863 44k INFO ====> Epoch: 2342, cost 14.00 s 2023-09-04 12:01:58,295 44k INFO Train Epoch: 2343 [81%] 2023-09-04 12:01:58,295 44k INFO Losses: [2.448472499847412, 2.3033697605133057, 7.868708610534668, 16.379730224609375, 0.602695643901825], step: 49200, lr: 7.46195015247753e-05, reference_loss: 29.602977752685547 2023-09-04 12:02:00,408 44k INFO ====> Epoch: 2343, cost 14.55 s 2023-09-04 12:02:14,533 44k INFO ====> Epoch: 2344, cost 14.12 s 2023-09-04 12:02:28,666 44k INFO ====> Epoch: 2345, cost 14.13 s 2023-09-04 12:02:42,811 44k INFO ====> Epoch: 2346, cost 14.14 s 2023-09-04 12:02:57,069 44k INFO ====> Epoch: 2347, cost 14.26 s 2023-09-04 12:03:11,299 44k INFO ====> Epoch: 2348, cost 14.23 s 2023-09-04 12:03:25,296 44k INFO ====> Epoch: 2349, cost 14.00 s 2023-09-04 12:03:39,400 44k INFO ====> Epoch: 2350, cost 14.10 s 2023-09-04 12:03:53,673 44k INFO ====> Epoch: 2351, cost 14.27 s 2023-09-04 12:04:07,755 44k INFO ====> Epoch: 2352, cost 14.08 s 2023-09-04 12:04:13,154 44k INFO Train Epoch: 2353 [33%] 2023-09-04 12:04:13,154 44k INFO Losses: [2.475799560546875, 2.300859212875366, 6.4525041580200195, 16.01068687438965, 0.583564043045044], step: 49400, lr: 7.452627959722119e-05, reference_loss: 27.823413848876953 2023-09-04 12:04:21,893 44k INFO ====> Epoch: 2353, cost 14.14 s 2023-09-04 12:04:36,241 44k INFO ====> Epoch: 2354, cost 14.35 s 2023-09-04 12:04:50,583 44k INFO ====> Epoch: 2355, cost 14.34 s 2023-09-04 12:05:04,770 44k INFO ====> Epoch: 2356, cost 14.19 s 2023-09-04 12:05:18,779 44k INFO ====> Epoch: 2357, cost 14.01 s 2023-09-04 12:05:32,618 44k INFO ====> Epoch: 2358, cost 13.84 s 2023-09-04 12:05:46,771 44k INFO ====> Epoch: 2359, cost 14.15 s 2023-09-04 12:06:01,103 44k INFO ====> Epoch: 2360, cost 14.33 s 2023-09-04 12:06:15,514 44k INFO ====> Epoch: 2361, cost 14.41 s 2023-09-04 12:06:28,507 44k INFO Train Epoch: 2362 [86%] 2023-09-04 12:06:28,507 44k INFO Losses: [2.139665365219116, 2.6565377712249756, 7.856541633605957, 16.47515296936035, 0.680456817150116], step: 49600, lr: 7.444247944148188e-05, reference_loss: 29.808353424072266 2023-09-04 12:06:33,166 44k INFO Saving model and optimizer state at iteration 2362 to ./logs\44k\G_49600.pth 2023-09-04 12:06:33,849 44k INFO Saving model and optimizer state at iteration 2362 to ./logs\44k\D_49600.pth 2023-09-04 12:06:34,944 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_47200.pth 2023-09-04 12:06:34,979 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_47200.pth 2023-09-04 12:06:36,078 44k INFO ====> Epoch: 2362, cost 20.56 s 2023-09-04 12:06:50,236 44k INFO ====> Epoch: 2363, cost 14.16 s 2023-09-04 12:07:04,379 44k INFO ====> Epoch: 2364, cost 14.14 s 2023-09-04 12:07:18,570 44k INFO ====> Epoch: 2365, cost 14.19 s 2023-09-04 12:07:32,637 44k INFO ====> Epoch: 2366, cost 14.07 s 2023-09-04 12:07:46,750 44k INFO ====> Epoch: 2367, cost 14.11 s 2023-09-04 12:08:01,066 44k INFO ====> Epoch: 2368, cost 14.32 s 2023-09-04 12:08:15,264 44k INFO ====> Epoch: 2369, cost 14.20 s 2023-09-04 12:08:29,339 44k INFO ====> Epoch: 2370, cost 14.07 s 2023-09-04 12:08:43,389 44k INFO ====> Epoch: 2371, cost 14.05 s 2023-09-04 12:08:49,628 44k INFO Train Epoch: 2372 [38%] 2023-09-04 12:08:49,628 44k INFO Losses: [2.280200242996216, 2.7640175819396973, 8.248690605163574, 16.526723861694336, 0.6121135950088501], step: 49800, lr: 7.434947866710472e-05, reference_loss: 30.431747436523438 2023-09-04 12:08:57,913 44k INFO ====> Epoch: 2372, cost 14.52 s 2023-09-04 12:09:12,195 44k INFO ====> Epoch: 2373, cost 14.28 s 2023-09-04 12:09:26,322 44k INFO ====> Epoch: 2374, cost 14.13 s 2023-09-04 12:09:40,329 44k INFO ====> Epoch: 2375, cost 14.01 s 2023-09-04 12:09:54,588 44k INFO ====> Epoch: 2376, cost 14.26 s 2023-09-04 12:10:08,728 44k INFO ====> Epoch: 2377, cost 14.14 s 2023-09-04 12:10:22,861 44k INFO ====> Epoch: 2378, cost 14.13 s 2023-09-04 12:10:36,979 44k INFO ====> Epoch: 2379, cost 14.12 s 2023-09-04 12:10:51,137 44k INFO ====> Epoch: 2380, cost 14.16 s 2023-09-04 12:11:04,882 44k INFO Train Epoch: 2381 [90%] 2023-09-04 12:11:04,882 44k INFO Losses: [2.3916120529174805, 2.4418892860412598, 7.149559497833252, 15.600994110107422, 0.5540425777435303], step: 50000, lr: 7.426587731299028e-05, reference_loss: 28.138097763061523 2023-09-04 12:11:05,620 44k INFO ====> Epoch: 2381, cost 14.48 s 2023-09-04 12:11:19,694 44k INFO ====> Epoch: 2382, cost 14.07 s 2023-09-04 12:11:33,713 44k INFO ====> Epoch: 2383, cost 14.02 s 2023-09-04 12:11:47,935 44k INFO ====> Epoch: 2384, cost 14.22 s 2023-09-04 12:12:02,368 44k INFO ====> Epoch: 2385, cost 14.43 s 2023-09-04 12:12:16,618 44k INFO ====> Epoch: 2386, cost 14.25 s 2023-09-04 12:12:30,668 44k INFO ====> Epoch: 2387, cost 14.05 s 2023-09-04 12:12:44,784 44k INFO ====> Epoch: 2388, cost 14.12 s 2023-09-04 12:12:59,048 44k INFO ====> Epoch: 2389, cost 14.26 s 2023-09-04 12:13:13,202 44k INFO ====> Epoch: 2390, cost 14.15 s 2023-09-04 12:13:19,883 44k INFO Train Epoch: 2391 [43%] 2023-09-04 12:13:19,883 44k INFO Losses: [2.2595057487487793, 2.610339641571045, 6.882870674133301, 15.59337043762207, 0.5695236921310425], step: 50200, lr: 7.417309716714176e-05, reference_loss: 27.91560935974121 2023-09-04 12:13:27,337 44k INFO ====> Epoch: 2391, cost 14.13 s 2023-09-04 12:13:41,549 44k INFO ====> Epoch: 2392, cost 14.21 s 2023-09-04 12:13:55,792 44k INFO ====> Epoch: 2393, cost 14.24 s 2023-09-04 12:14:10,183 44k INFO ====> Epoch: 2394, cost 14.39 s 2023-09-04 12:14:24,344 44k INFO ====> Epoch: 2395, cost 14.16 s 2023-09-04 12:14:38,451 44k INFO ====> Epoch: 2396, cost 14.11 s 2023-09-04 12:14:52,608 44k INFO ====> Epoch: 2397, cost 14.16 s 2023-09-04 12:15:06,754 44k INFO ====> Epoch: 2398, cost 14.15 s 2023-09-04 12:15:20,840 44k INFO ====> Epoch: 2399, cost 14.09 s 2023-09-04 12:15:34,751 44k INFO Train Epoch: 2400 [95%] 2023-09-04 12:15:34,751 44k INFO Losses: [2.286947250366211, 2.5406906604766846, 7.6594953536987305, 16.84134864807129, 0.33785900473594666], step: 50400, lr: 7.408969414302911e-05, reference_loss: 29.666339874267578 2023-09-04 12:15:39,523 44k INFO Saving model and optimizer state at iteration 2400 to ./logs\44k\G_50400.pth 2023-09-04 12:15:40,217 44k INFO Saving model and optimizer state at iteration 2400 to ./logs\44k\D_50400.pth 2023-09-04 12:15:41,411 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_48000.pth 2023-09-04 12:15:41,443 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_48000.pth 2023-09-04 12:15:41,444 44k INFO ====> Epoch: 2400, cost 20.60 s 2023-09-04 12:15:55,565 44k INFO ====> Epoch: 2401, cost 14.12 s 2023-09-04 12:16:09,771 44k INFO ====> Epoch: 2402, cost 14.21 s 2023-09-04 12:16:23,730 44k INFO ====> Epoch: 2403, cost 13.96 s 2023-09-04 12:16:37,859 44k INFO ====> Epoch: 2404, cost 14.13 s 2023-09-04 12:16:52,068 44k INFO ====> Epoch: 2405, cost 14.21 s 2023-09-04 12:17:06,354 44k INFO ====> Epoch: 2406, cost 14.29 s 2023-09-04 12:17:20,600 44k INFO ====> Epoch: 2407, cost 14.25 s 2023-09-04 12:17:34,472 44k INFO ====> Epoch: 2408, cost 13.87 s 2023-09-04 12:17:48,810 44k INFO ====> Epoch: 2409, cost 14.34 s 2023-09-04 12:17:56,535 44k INFO Train Epoch: 2410 [48%] 2023-09-04 12:17:56,535 44k INFO Losses: [2.400784492492676, 2.305004358291626, 7.902055263519287, 16.81089210510254, 0.7868520021438599], step: 50600, lr: 7.399713410230553e-05, reference_loss: 30.20558738708496 2023-09-04 12:18:03,522 44k INFO ====> Epoch: 2410, cost 14.71 s 2023-09-04 12:18:17,785 44k INFO ====> Epoch: 2411, cost 14.26 s 2023-09-04 12:18:31,982 44k INFO ====> Epoch: 2412, cost 14.20 s 2023-09-04 12:18:46,047 44k INFO ====> Epoch: 2413, cost 14.06 s 2023-09-04 12:19:00,338 44k INFO ====> Epoch: 2414, cost 14.29 s 2023-09-04 12:19:14,682 44k INFO ====> Epoch: 2415, cost 14.34 s 2023-09-04 12:19:28,626 44k INFO ====> Epoch: 2416, cost 13.94 s 2023-09-04 12:19:42,659 44k INFO ====> Epoch: 2417, cost 14.03 s 2023-09-04 12:19:56,894 44k INFO ====> Epoch: 2418, cost 14.24 s 2023-09-04 12:20:11,183 44k INFO ====> Epoch: 2419, cost 14.29 s 2023-09-04 12:20:11,891 44k INFO Train Epoch: 2420 [0%] 2023-09-04 12:20:11,891 44k INFO Losses: [2.549985885620117, 2.2085647583007812, 6.394765377044678, 13.721134185791016, 0.6824530363082886], step: 50800, lr: 7.390468969657324e-05, reference_loss: 25.556903839111328 2023-09-04 12:20:25,470 44k INFO ====> Epoch: 2420, cost 14.29 s 2023-09-04 12:20:39,706 44k INFO ====> Epoch: 2421, cost 14.24 s 2023-09-04 12:20:53,942 44k INFO ====> Epoch: 2422, cost 14.24 s 2023-09-04 12:21:08,136 44k INFO ====> Epoch: 2423, cost 14.19 s 2023-09-04 12:21:22,221 44k INFO ====> Epoch: 2424, cost 14.09 s 2023-09-04 12:21:36,317 44k INFO ====> Epoch: 2425, cost 14.10 s 2023-09-04 12:21:50,622 44k INFO ====> Epoch: 2426, cost 14.30 s 2023-09-04 12:22:04,707 44k INFO ====> Epoch: 2427, cost 14.08 s 2023-09-04 12:22:18,855 44k INFO ====> Epoch: 2428, cost 14.15 s 2023-09-04 12:22:27,121 44k INFO Train Epoch: 2429 [52%] 2023-09-04 12:22:27,121 44k INFO Losses: [2.5105082988739014, 2.294558048248291, 7.219801425933838, 16.65176010131836, 0.4825359880924225], step: 51000, lr: 7.382158847992981e-05, reference_loss: 29.159164428710938 2023-09-04 12:22:33,329 44k INFO ====> Epoch: 2429, cost 14.47 s 2023-09-04 12:22:47,521 44k INFO ====> Epoch: 2430, cost 14.19 s 2023-09-04 12:23:01,806 44k INFO ====> Epoch: 2431, cost 14.28 s 2023-09-04 12:23:15,916 44k INFO ====> Epoch: 2432, cost 14.11 s 2023-09-04 12:23:29,917 44k INFO ====> Epoch: 2433, cost 14.00 s 2023-09-04 12:23:44,089 44k INFO ====> Epoch: 2434, cost 14.17 s 2023-09-04 12:23:58,615 44k INFO ====> Epoch: 2435, cost 14.53 s 2023-09-04 12:24:12,891 44k INFO ====> Epoch: 2436, cost 14.28 s 2023-09-04 12:24:27,074 44k INFO ====> Epoch: 2437, cost 14.18 s 2023-09-04 12:24:41,187 44k INFO ====> Epoch: 2438, cost 14.11 s 2023-09-04 12:24:42,558 44k INFO Train Epoch: 2439 [5%] 2023-09-04 12:24:42,559 44k INFO Losses: [2.500767469406128, 2.376901626586914, 6.839091777801514, 15.616488456726074, 0.6278927326202393], step: 51200, lr: 7.372936338283611e-05, reference_loss: 27.96114158630371 2023-09-04 12:24:47,281 44k INFO Saving model and optimizer state at iteration 2439 to ./logs\44k\G_51200.pth 2023-09-04 12:24:47,957 44k INFO Saving model and optimizer state at iteration 2439 to ./logs\44k\D_51200.pth 2023-09-04 12:24:49,022 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_48800.pth 2023-09-04 12:24:49,055 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_48800.pth 2023-09-04 12:25:02,009 44k INFO ====> Epoch: 2439, cost 20.82 s 2023-09-04 12:25:16,413 44k INFO ====> Epoch: 2440, cost 14.40 s 2023-09-04 12:25:30,450 44k INFO ====> Epoch: 2441, cost 14.04 s 2023-09-04 12:25:44,495 44k INFO ====> Epoch: 2442, cost 14.05 s 2023-09-04 12:25:58,856 44k INFO ====> Epoch: 2443, cost 14.36 s 2023-09-04 12:26:13,214 44k INFO ====> Epoch: 2444, cost 14.36 s 2023-09-04 12:26:27,396 44k INFO ====> Epoch: 2445, cost 14.18 s 2023-09-04 12:26:41,423 44k INFO ====> Epoch: 2446, cost 14.03 s 2023-09-04 12:26:55,694 44k INFO ====> Epoch: 2447, cost 14.27 s 2023-09-04 12:27:04,824 44k INFO Train Epoch: 2448 [57%] 2023-09-04 12:27:04,824 44k INFO Losses: [2.2388017177581787, 2.4845683574676514, 6.956584930419922, 15.303627967834473, 0.6282182931900024], step: 51400, lr: 7.364645930970334e-05, reference_loss: 27.611799240112305 2023-09-04 12:27:10,344 44k INFO ====> Epoch: 2448, cost 14.65 s 2023-09-04 12:27:24,316 44k INFO ====> Epoch: 2449, cost 13.97 s 2023-09-04 12:27:38,233 44k INFO ====> Epoch: 2450, cost 13.92 s 2023-09-04 12:27:52,257 44k INFO ====> Epoch: 2451, cost 14.02 s 2023-09-04 12:28:06,394 44k INFO ====> Epoch: 2452, cost 14.14 s 2023-09-04 12:28:20,441 44k INFO ====> Epoch: 2453, cost 14.05 s 2023-09-04 12:28:34,547 44k INFO ====> Epoch: 2454, cost 14.11 s 2023-09-04 12:28:48,883 44k INFO ====> Epoch: 2455, cost 14.34 s 2023-09-04 12:29:03,164 44k INFO ====> Epoch: 2456, cost 14.28 s 2023-09-04 12:29:17,123 44k INFO ====> Epoch: 2457, cost 13.96 s 2023-09-04 12:29:19,109 44k INFO Train Epoch: 2458 [10%] 2023-09-04 12:29:19,110 44k INFO Losses: [2.4564905166625977, 2.507822036743164, 7.037895202636719, 16.547405242919922, 0.5921065211296082], step: 51600, lr: 7.355445300097575e-05, reference_loss: 29.141719818115234 2023-09-04 12:29:31,309 44k INFO ====> Epoch: 2458, cost 14.19 s 2023-09-04 12:29:45,558 44k INFO ====> Epoch: 2459, cost 14.25 s 2023-09-04 12:29:59,975 44k INFO ====> Epoch: 2460, cost 14.42 s 2023-09-04 12:30:14,301 44k INFO ====> Epoch: 2461, cost 14.33 s 2023-09-04 12:30:28,345 44k INFO ====> Epoch: 2462, cost 14.04 s 2023-09-04 12:30:42,539 44k INFO ====> Epoch: 2463, cost 14.19 s 2023-09-04 12:30:56,867 44k INFO ====> Epoch: 2464, cost 14.33 s 2023-09-04 12:31:11,283 44k INFO ====> Epoch: 2465, cost 14.42 s 2023-09-04 12:31:25,339 44k INFO ====> Epoch: 2466, cost 14.06 s 2023-09-04 12:31:34,870 44k INFO Train Epoch: 2467 [62%] 2023-09-04 12:31:34,870 44k INFO Losses: [2.227658748626709, 2.639375925064087, 8.552595138549805, 15.989373207092285, 0.5428081154823303], step: 51800, lr: 7.347174560366418e-05, reference_loss: 29.951812744140625 2023-09-04 12:31:39,571 44k INFO ====> Epoch: 2467, cost 14.23 s 2023-09-04 12:31:53,878 44k INFO ====> Epoch: 2468, cost 14.31 s 2023-09-04 12:32:08,140 44k INFO ====> Epoch: 2469, cost 14.26 s 2023-09-04 12:32:22,458 44k INFO ====> Epoch: 2470, cost 14.32 s 2023-09-04 12:32:36,651 44k INFO ====> Epoch: 2471, cost 14.19 s 2023-09-04 12:32:50,891 44k INFO ====> Epoch: 2472, cost 14.24 s 2023-09-04 12:33:05,154 44k INFO ====> Epoch: 2473, cost 14.26 s 2023-09-04 12:33:19,164 44k INFO ====> Epoch: 2474, cost 14.01 s 2023-09-04 12:33:33,082 44k INFO ====> Epoch: 2475, cost 13.92 s 2023-09-04 12:33:47,280 44k INFO ====> Epoch: 2476, cost 14.20 s 2023-09-04 12:33:50,010 44k INFO Train Epoch: 2477 [14%] 2023-09-04 12:33:50,010 44k INFO Losses: [2.412935495376587, 2.393038749694824, 8.489027976989746, 16.307390213012695, 0.6546619534492493], step: 52000, lr: 7.337995756426454e-05, reference_loss: 30.25705337524414 2023-09-04 12:33:54,735 44k INFO Saving model and optimizer state at iteration 2477 to ./logs\44k\G_52000.pth 2023-09-04 12:33:55,395 44k INFO Saving model and optimizer state at iteration 2477 to ./logs\44k\D_52000.pth 2023-09-04 12:33:56,511 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_49600.pth 2023-09-04 12:33:56,543 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_49600.pth 2023-09-04 12:34:07,973 44k INFO ====> Epoch: 2477, cost 20.69 s 2023-09-04 12:34:21,980 44k INFO ====> Epoch: 2478, cost 14.01 s 2023-09-04 12:34:36,056 44k INFO ====> Epoch: 2479, cost 14.08 s 2023-09-04 12:34:50,291 44k INFO ====> Epoch: 2480, cost 14.24 s 2023-09-04 12:35:04,760 44k INFO ====> Epoch: 2481, cost 14.47 s 2023-09-04 12:35:18,921 44k INFO ====> Epoch: 2482, cost 14.16 s 2023-09-04 12:35:32,918 44k INFO ====> Epoch: 2483, cost 14.00 s 2023-09-04 12:35:47,216 44k INFO ====> Epoch: 2484, cost 14.30 s 2023-09-04 12:36:01,603 44k INFO ====> Epoch: 2485, cost 14.39 s 2023-09-04 12:36:12,042 44k INFO Train Epoch: 2486 [67%] 2023-09-04 12:36:12,043 44k INFO Losses: [2.0799717903137207, 2.8534698486328125, 8.65933609008789, 15.502638816833496, 0.5895016193389893], step: 52200, lr: 7.32974463761942e-05, reference_loss: 29.684919357299805 2023-09-04 12:36:16,100 44k INFO ====> Epoch: 2486, cost 14.50 s 2023-09-04 12:36:30,221 44k INFO ====> Epoch: 2487, cost 14.12 s 2023-09-04 12:36:44,475 44k INFO ====> Epoch: 2488, cost 14.25 s 2023-09-04 12:36:58,977 44k INFO ====> Epoch: 2489, cost 14.50 s 2023-09-04 12:37:13,183 44k INFO ====> Epoch: 2490, cost 14.21 s 2023-09-04 12:37:27,215 44k INFO ====> Epoch: 2491, cost 14.03 s 2023-09-04 12:37:41,237 44k INFO ====> Epoch: 2492, cost 14.02 s 2023-09-04 12:37:55,493 44k INFO ====> Epoch: 2493, cost 14.26 s 2023-09-04 12:38:09,823 44k INFO ====> Epoch: 2494, cost 14.33 s 2023-09-04 12:38:23,943 44k INFO ====> Epoch: 2495, cost 14.12 s 2023-09-04 12:38:27,349 44k INFO Train Epoch: 2496 [19%] 2023-09-04 12:38:27,349 44k INFO Losses: [2.4756364822387695, 2.2719430923461914, 7.076128959655762, 16.586517333984375, 0.582339346408844], step: 52400, lr: 7.320587608831558e-05, reference_loss: 28.992565155029297 2023-09-04 12:38:38,413 44k INFO ====> Epoch: 2496, cost 14.47 s 2023-09-04 12:38:52,580 44k INFO ====> Epoch: 2497, cost 14.17 s 2023-09-04 12:39:07,079 44k INFO ====> Epoch: 2498, cost 14.50 s 2023-09-04 12:39:21,056 44k INFO ====> Epoch: 2499, cost 13.98 s 2023-09-04 12:39:35,007 44k INFO ====> Epoch: 2500, cost 13.95 s 2023-09-04 12:39:49,184 44k INFO ====> Epoch: 2501, cost 14.18 s 2023-09-04 12:40:03,461 44k INFO ====> Epoch: 2502, cost 14.28 s 2023-09-04 12:40:17,744 44k INFO ====> Epoch: 2503, cost 14.28 s 2023-09-04 12:40:31,729 44k INFO ====> Epoch: 2504, cost 13.99 s 2023-09-04 12:40:42,632 44k INFO Train Epoch: 2505 [71%] 2023-09-04 12:40:42,632 44k INFO Losses: [2.469583511352539, 2.3794500827789307, 7.285552024841309, 15.049098014831543, 0.4544709622859955], step: 52600, lr: 7.312356064401339e-05, reference_loss: 27.638154983520508 2023-09-04 12:40:46,049 44k INFO ====> Epoch: 2505, cost 14.32 s 2023-09-04 12:41:00,259 44k INFO ====> Epoch: 2506, cost 14.21 s 2023-09-04 12:41:14,447 44k INFO ====> Epoch: 2507, cost 14.19 s 2023-09-04 12:41:28,488 44k INFO ====> Epoch: 2508, cost 14.04 s 2023-09-04 12:41:42,630 44k INFO ====> Epoch: 2509, cost 14.14 s 2023-09-04 12:41:56,896 44k INFO ====> Epoch: 2510, cost 14.27 s 2023-09-04 12:42:11,128 44k INFO ====> Epoch: 2511, cost 14.23 s 2023-09-04 12:42:25,226 44k INFO ====> Epoch: 2512, cost 14.10 s 2023-09-04 12:42:39,464 44k INFO ====> Epoch: 2513, cost 14.24 s 2023-09-04 12:42:53,741 44k INFO ====> Epoch: 2514, cost 14.28 s 2023-09-04 12:42:57,879 44k INFO Train Epoch: 2515 [24%] 2023-09-04 12:42:57,879 44k INFO Losses: [2.4435548782348633, 2.235154867172241, 6.354360103607178, 15.428868293762207, 0.7487209439277649], step: 52800, lr: 7.303220759107733e-05, reference_loss: 27.210657119750977 2023-09-04 12:43:02,656 44k INFO Saving model and optimizer state at iteration 2515 to ./logs\44k\G_52800.pth 2023-09-04 12:43:03,228 44k INFO Saving model and optimizer state at iteration 2515 to ./logs\44k\D_52800.pth 2023-09-04 12:43:04,818 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_50400.pth 2023-09-04 12:43:04,852 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_50400.pth 2023-09-04 12:43:14,912 44k INFO ====> Epoch: 2515, cost 21.17 s 2023-09-04 12:43:28,861 44k INFO ====> Epoch: 2516, cost 13.95 s 2023-09-04 12:43:42,712 44k INFO ====> Epoch: 2517, cost 13.85 s 2023-09-04 12:43:56,991 44k INFO ====> Epoch: 2518, cost 14.28 s 2023-09-04 12:44:11,296 44k INFO ====> Epoch: 2519, cost 14.31 s 2023-09-04 12:44:25,472 44k INFO ====> Epoch: 2520, cost 14.18 s 2023-09-04 12:44:39,526 44k INFO ====> Epoch: 2521, cost 14.05 s 2023-09-04 12:44:53,695 44k INFO ====> Epoch: 2522, cost 14.17 s 2023-09-04 12:45:07,922 44k INFO ====> Epoch: 2523, cost 14.23 s 2023-09-04 12:45:19,464 44k INFO Train Epoch: 2524 [76%] 2023-09-04 12:45:19,465 44k INFO Losses: [2.096515417098999, 2.7527527809143066, 8.217738151550293, 17.24620819091797, 0.8012366890907288], step: 53000, lr: 7.29500874261745e-05, reference_loss: 31.114450454711914 2023-09-04 12:45:22,202 44k INFO ====> Epoch: 2524, cost 14.28 s 2023-09-04 12:45:36,154 44k INFO ====> Epoch: 2525, cost 13.95 s 2023-09-04 12:45:50,135 44k INFO ====> Epoch: 2526, cost 13.98 s 2023-09-04 12:46:04,305 44k INFO ====> Epoch: 2527, cost 14.17 s 2023-09-04 12:46:18,229 44k INFO ====> Epoch: 2528, cost 13.92 s 2023-09-04 12:46:32,277 44k INFO ====> Epoch: 2529, cost 14.05 s 2023-09-04 12:46:46,503 44k INFO ====> Epoch: 2530, cost 14.23 s 2023-09-04 12:47:00,831 44k INFO ====> Epoch: 2531, cost 14.33 s 2023-09-04 12:47:15,012 44k INFO ====> Epoch: 2532, cost 14.18 s 2023-09-04 12:47:29,079 44k INFO ====> Epoch: 2533, cost 14.07 s 2023-09-04 12:47:33,894 44k INFO Train Epoch: 2534 [29%] 2023-09-04 12:47:33,895 44k INFO Losses: [2.4518706798553467, 2.4546091556549072, 7.246434688568115, 16.076282501220703, 0.5902511477470398], step: 53200, lr: 7.285895109282803e-05, reference_loss: 28.819448471069336 2023-09-04 12:47:43,530 44k INFO ====> Epoch: 2534, cost 14.45 s 2023-09-04 12:47:57,796 44k INFO ====> Epoch: 2535, cost 14.27 s 2023-09-04 12:48:12,252 44k INFO ====> Epoch: 2536, cost 14.46 s 2023-09-04 12:48:26,563 44k INFO ====> Epoch: 2537, cost 14.31 s 2023-09-04 12:48:40,722 44k INFO ====> Epoch: 2538, cost 14.16 s 2023-09-04 12:48:55,005 44k INFO ====> Epoch: 2539, cost 14.28 s 2023-09-04 12:49:09,274 44k INFO ====> Epoch: 2540, cost 14.27 s 2023-09-04 12:49:23,337 44k INFO ====> Epoch: 2541, cost 14.06 s 2023-09-04 12:49:37,328 44k INFO ====> Epoch: 2542, cost 13.99 s 2023-09-04 12:49:49,796 44k INFO Train Epoch: 2543 [81%] 2023-09-04 12:49:49,797 44k INFO Losses: [2.265263557434082, 2.5381698608398438, 7.247192859649658, 15.765625953674316, 0.7704932689666748], step: 53400, lr: 7.277702574405738e-05, reference_loss: 28.586746215820312 2023-09-04 12:49:51,847 44k INFO ====> Epoch: 2543, cost 14.52 s 2023-09-04 12:50:06,073 44k INFO ====> Epoch: 2544, cost 14.23 s 2023-09-04 12:50:20,362 44k INFO ====> Epoch: 2545, cost 14.29 s 2023-09-04 12:50:34,515 44k INFO ====> Epoch: 2546, cost 14.15 s 2023-09-04 12:50:48,703 44k INFO ====> Epoch: 2547, cost 14.19 s 2023-09-04 12:51:03,017 44k INFO ====> Epoch: 2548, cost 14.31 s 2023-09-04 12:51:17,060 44k INFO ====> Epoch: 2549, cost 14.04 s 2023-09-04 12:51:31,023 44k INFO ====> Epoch: 2550, cost 13.96 s 2023-09-04 12:51:45,013 44k INFO ====> Epoch: 2551, cost 13.99 s 2023-09-04 12:51:59,166 44k INFO ====> Epoch: 2552, cost 14.15 s 2023-09-04 12:52:04,683 44k INFO Train Epoch: 2553 [33%] 2023-09-04 12:52:04,683 44k INFO Losses: [2.4578053951263428, 2.2995100021362305, 6.1631903648376465, 15.569135665893555, 0.6478917598724365], step: 53600, lr: 7.268610561617012e-05, reference_loss: 27.13753318786621 2023-09-04 12:52:09,368 44k INFO Saving model and optimizer state at iteration 2553 to ./logs\44k\G_53600.pth 2023-09-04 12:52:09,930 44k INFO Saving model and optimizer state at iteration 2553 to ./logs\44k\D_53600.pth 2023-09-04 12:52:11,563 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_51200.pth 2023-09-04 12:52:11,607 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_51200.pth 2023-09-04 12:52:20,168 44k INFO ====> Epoch: 2553, cost 21.00 s 2023-09-04 12:52:34,071 44k INFO ====> Epoch: 2554, cost 13.90 s 2023-09-04 12:52:48,261 44k INFO ====> Epoch: 2555, cost 14.19 s 2023-09-04 12:53:02,569 44k INFO ====> Epoch: 2556, cost 14.31 s 2023-09-04 12:53:16,598 44k INFO ====> Epoch: 2557, cost 14.03 s 2023-09-04 12:53:30,704 44k INFO ====> Epoch: 2558, cost 14.11 s 2023-09-04 12:53:44,931 44k INFO ====> Epoch: 2559, cost 14.23 s 2023-09-04 12:53:59,309 44k INFO ====> Epoch: 2560, cost 14.38 s 2023-09-04 12:54:13,692 44k INFO ====> Epoch: 2561, cost 14.38 s 2023-09-04 12:54:26,689 44k INFO Train Epoch: 2562 [86%] 2023-09-04 12:54:26,689 44k INFO Losses: [2.333936929702759, 2.5397753715515137, 8.753963470458984, 15.96527099609375, 0.7505355477333069], step: 53800, lr: 7.260437462136348e-05, reference_loss: 30.343482971191406 2023-09-04 12:54:28,003 44k INFO ====> Epoch: 2562, cost 14.31 s 2023-09-04 12:54:42,068 44k INFO ====> Epoch: 2563, cost 14.07 s 2023-09-04 12:54:56,307 44k INFO ====> Epoch: 2564, cost 14.24 s 2023-09-04 12:55:10,546 44k INFO ====> Epoch: 2565, cost 14.24 s 2023-09-04 12:55:24,519 44k INFO ====> Epoch: 2566, cost 13.97 s 2023-09-04 12:55:38,743 44k INFO ====> Epoch: 2567, cost 14.22 s 2023-09-04 12:55:52,932 44k INFO ====> Epoch: 2568, cost 14.19 s 2023-09-04 12:56:07,384 44k INFO ====> Epoch: 2569, cost 14.45 s 2023-09-04 12:56:21,591 44k INFO ====> Epoch: 2570, cost 14.21 s 2023-09-04 12:56:35,619 44k INFO ====> Epoch: 2571, cost 14.03 s 2023-09-04 12:56:41,784 44k INFO Train Epoch: 2572 [38%] 2023-09-04 12:56:41,785 44k INFO Losses: [2.1594250202178955, 2.8270108699798584, 7.80267333984375, 16.78512191772461, 0.5826456546783447], step: 54000, lr: 7.251367018602473e-05, reference_loss: 30.156875610351562 2023-09-04 12:56:50,041 44k INFO ====> Epoch: 2572, cost 14.42 s 2023-09-04 12:57:04,332 44k INFO ====> Epoch: 2573, cost 14.29 s 2023-09-04 12:57:18,414 44k INFO ====> Epoch: 2574, cost 14.08 s 2023-09-04 12:57:32,386 44k INFO ====> Epoch: 2575, cost 13.97 s 2023-09-04 12:57:46,437 44k INFO ====> Epoch: 2576, cost 14.05 s 2023-09-04 12:58:00,620 44k INFO ====> Epoch: 2577, cost 14.18 s 2023-09-04 12:58:14,764 44k INFO ====> Epoch: 2578, cost 14.14 s 2023-09-04 12:58:28,741 44k INFO ====> Epoch: 2579, cost 13.98 s 2023-09-04 12:58:42,801 44k INFO ====> Epoch: 2580, cost 14.06 s 2023-09-04 12:58:56,626 44k INFO Train Epoch: 2581 [90%] 2023-09-04 12:58:56,627 44k INFO Losses: [2.291651487350464, 2.5714316368103027, 9.047813415527344, 18.180503845214844, 0.4712972342967987], step: 54200, lr: 7.243213308411036e-05, reference_loss: 32.56269836425781 2023-09-04 12:58:57,317 44k INFO ====> Epoch: 2581, cost 14.52 s 2023-09-04 12:59:11,654 44k INFO ====> Epoch: 2582, cost 14.34 s 2023-09-04 12:59:25,765 44k INFO ====> Epoch: 2583, cost 14.11 s 2023-09-04 12:59:39,847 44k INFO ====> Epoch: 2584, cost 14.08 s 2023-09-04 12:59:54,247 44k INFO ====> Epoch: 2585, cost 14.40 s 2023-09-04 13:00:08,653 44k INFO ====> Epoch: 2586, cost 14.41 s 2023-09-04 13:00:22,700 44k INFO ====> Epoch: 2587, cost 14.05 s 2023-09-04 13:00:36,809 44k INFO ====> Epoch: 2588, cost 14.11 s 2023-09-04 13:00:51,051 44k INFO ====> Epoch: 2589, cost 14.24 s 2023-09-04 13:01:05,290 44k INFO ====> Epoch: 2590, cost 14.24 s 2023-09-04 13:01:12,254 44k INFO Train Epoch: 2591 [43%] 2023-09-04 13:01:12,254 44k INFO Losses: [2.4355368614196777, 2.46673583984375, 6.876979351043701, 14.884655952453613, 0.723854660987854], step: 54400, lr: 7.23416438296262e-05, reference_loss: 27.38776206970215 2023-09-04 13:01:16,923 44k INFO Saving model and optimizer state at iteration 2591 to ./logs\44k\G_54400.pth 2023-09-04 13:01:17,645 44k INFO Saving model and optimizer state at iteration 2591 to ./logs\44k\D_54400.pth 2023-09-04 13:01:18,742 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_52000.pth 2023-09-04 13:01:18,776 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_52000.pth 2023-09-04 13:01:26,047 44k INFO ====> Epoch: 2591, cost 20.76 s 2023-09-04 13:01:40,077 44k INFO ====> Epoch: 2592, cost 14.03 s 2023-09-04 13:01:54,294 44k INFO ====> Epoch: 2593, cost 14.22 s 2023-09-04 13:02:08,590 44k INFO ====> Epoch: 2594, cost 14.30 s 2023-09-04 13:02:22,818 44k INFO ====> Epoch: 2595, cost 14.23 s 2023-09-04 13:02:36,835 44k INFO ====> Epoch: 2596, cost 14.02 s 2023-09-04 13:02:50,951 44k INFO ====> Epoch: 2597, cost 14.12 s 2023-09-04 13:03:05,183 44k INFO ====> Epoch: 2598, cost 14.23 s 2023-09-04 13:03:19,214 44k INFO ====> Epoch: 2599, cost 14.03 s 2023-09-04 13:03:33,362 44k INFO Train Epoch: 2600 [95%] 2023-09-04 13:03:33,363 44k INFO Losses: [2.2844977378845215, 2.388699769973755, 7.796229362487793, 16.776514053344727, 0.5846626162528992], step: 54600, lr: 7.226030016062616e-05, reference_loss: 29.830604553222656 2023-09-04 13:03:33,613 44k INFO ====> Epoch: 2600, cost 14.40 s 2023-09-04 13:03:47,504 44k INFO ====> Epoch: 2601, cost 13.89 s 2023-09-04 13:04:01,704 44k INFO ====> Epoch: 2602, cost 14.20 s 2023-09-04 13:04:15,939 44k INFO ====> Epoch: 2603, cost 14.23 s 2023-09-04 13:04:30,033 44k INFO ====> Epoch: 2604, cost 14.09 s 2023-09-04 13:04:44,231 44k INFO ====> Epoch: 2605, cost 14.20 s 2023-09-04 13:04:58,437 44k INFO ====> Epoch: 2606, cost 14.21 s 2023-09-04 13:05:12,567 44k INFO ====> Epoch: 2607, cost 14.13 s 2023-09-04 13:05:26,558 44k INFO ====> Epoch: 2608, cost 13.99 s 2023-09-04 13:05:40,591 44k INFO ====> Epoch: 2609, cost 14.03 s 2023-09-04 13:05:48,191 44k INFO Train Epoch: 2610 [48%] 2023-09-04 13:05:48,191 44k INFO Losses: [2.3814449310302734, 2.569826126098633, 10.27196216583252, 18.116104125976562, 0.6945674419403076], step: 54800, lr: 7.217002557651661e-05, reference_loss: 34.033905029296875 2023-09-04 13:05:55,217 44k INFO ====> Epoch: 2610, cost 14.63 s 2023-09-04 13:06:09,569 44k INFO ====> Epoch: 2611, cost 14.35 s 2023-09-04 13:06:23,717 44k INFO ====> Epoch: 2612, cost 14.15 s 2023-09-04 13:06:37,756 44k INFO ====> Epoch: 2613, cost 14.04 s 2023-09-04 13:06:51,941 44k INFO ====> Epoch: 2614, cost 14.18 s 2023-09-04 13:07:06,304 44k INFO ====> Epoch: 2615, cost 14.36 s 2023-09-04 13:07:20,319 44k INFO ====> Epoch: 2616, cost 14.01 s 2023-09-04 13:07:34,283 44k INFO ====> Epoch: 2617, cost 13.96 s 2023-09-04 13:07:48,689 44k INFO ====> Epoch: 2618, cost 14.41 s 2023-09-04 13:08:03,052 44k INFO ====> Epoch: 2619, cost 14.36 s 2023-09-04 13:08:03,759 44k INFO Train Epoch: 2620 [0%] 2023-09-04 13:08:03,760 44k INFO Losses: [2.0078980922698975, 3.174234628677368, 8.65657901763916, 15.326498031616211, 0.7344678640365601], step: 55000, lr: 7.207986377218403e-05, reference_loss: 29.89967918395996 2023-09-04 13:08:17,600 44k INFO ====> Epoch: 2620, cost 14.55 s 2023-09-04 13:08:31,767 44k INFO ====> Epoch: 2621, cost 14.17 s 2023-09-04 13:08:45,883 44k INFO ====> Epoch: 2622, cost 14.12 s 2023-09-04 13:09:00,051 44k INFO ====> Epoch: 2623, cost 14.17 s 2023-09-04 13:09:14,231 44k INFO ====> Epoch: 2624, cost 14.18 s 2023-09-04 13:09:28,182 44k INFO ====> Epoch: 2625, cost 13.95 s 2023-09-04 13:09:42,315 44k INFO ====> Epoch: 2626, cost 14.13 s 2023-09-04 13:09:56,512 44k INFO ====> Epoch: 2627, cost 14.20 s 2023-09-04 13:10:10,930 44k INFO ====> Epoch: 2628, cost 14.42 s 2023-09-04 13:10:19,125 44k INFO Train Epoch: 2629 [52%] 2023-09-04 13:10:19,125 44k INFO Losses: [2.259956121444702, 2.583770513534546, 8.138216018676758, 16.44722557067871, 0.6660944223403931], step: 55200, lr: 7.199881445854027e-05, reference_loss: 30.09526252746582 2023-09-04 13:10:23,792 44k INFO Saving model and optimizer state at iteration 2629 to ./logs\44k\G_55200.pth 2023-09-04 13:10:24,397 44k INFO Saving model and optimizer state at iteration 2629 to ./logs\44k\D_55200.pth 2023-09-04 13:10:25,550 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_52800.pth 2023-09-04 13:10:25,587 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_52800.pth 2023-09-04 13:10:31,518 44k INFO ====> Epoch: 2629, cost 20.59 s 2023-09-04 13:10:45,660 44k INFO ====> Epoch: 2630, cost 14.14 s 2023-09-04 13:10:59,866 44k INFO ====> Epoch: 2631, cost 14.21 s 2023-09-04 13:11:14,098 44k INFO ====> Epoch: 2632, cost 14.23 s 2023-09-04 13:11:28,312 44k INFO ====> Epoch: 2633, cost 14.21 s 2023-09-04 13:11:42,276 44k INFO ====> Epoch: 2634, cost 13.96 s 2023-09-04 13:11:56,618 44k INFO ====> Epoch: 2635, cost 14.34 s 2023-09-04 13:12:11,023 44k INFO ====> Epoch: 2636, cost 14.40 s 2023-09-04 13:12:25,246 44k INFO ====> Epoch: 2637, cost 14.22 s 2023-09-04 13:12:39,469 44k INFO ====> Epoch: 2638, cost 14.22 s 2023-09-04 13:12:40,837 44k INFO Train Epoch: 2639 [5%] 2023-09-04 13:12:40,838 44k INFO Losses: [2.244086742401123, 2.7096221446990967, 6.981701374053955, 14.596410751342773, 0.6790992617607117], step: 55400, lr: 7.190886654776244e-05, reference_loss: 27.210920333862305 2023-09-04 13:12:53,939 44k INFO ====> Epoch: 2639, cost 14.47 s 2023-09-04 13:13:08,271 44k INFO ====> Epoch: 2640, cost 14.33 s 2023-09-04 13:13:22,102 44k INFO ====> Epoch: 2641, cost 13.83 s 2023-09-04 13:13:36,076 44k INFO ====> Epoch: 2642, cost 13.97 s 2023-09-04 13:13:50,345 44k INFO ====> Epoch: 2643, cost 14.27 s 2023-09-04 13:14:04,620 44k INFO ====> Epoch: 2644, cost 14.28 s 2023-09-04 13:14:18,744 44k INFO ====> Epoch: 2645, cost 14.12 s 2023-09-04 13:14:32,819 44k INFO ====> Epoch: 2646, cost 14.08 s 2023-09-04 13:14:46,869 44k INFO ====> Epoch: 2647, cost 14.05 s 2023-09-04 13:14:55,771 44k INFO Train Epoch: 2648 [57%] 2023-09-04 13:14:55,772 44k INFO Losses: [2.2334296703338623, 2.601607084274292, 8.250819206237793, 16.84927749633789, 0.5633261799812317], step: 55600, lr: 7.182800950983827e-05, reference_loss: 30.498458862304688 2023-09-04 13:15:01,214 44k INFO ====> Epoch: 2648, cost 14.34 s 2023-09-04 13:15:15,488 44k INFO ====> Epoch: 2649, cost 14.27 s 2023-09-04 13:15:29,489 44k INFO ====> Epoch: 2650, cost 14.00 s 2023-09-04 13:15:43,490 44k INFO ====> Epoch: 2651, cost 14.00 s 2023-09-04 13:15:57,702 44k INFO ====> Epoch: 2652, cost 14.21 s 2023-09-04 13:16:11,795 44k INFO ====> Epoch: 2653, cost 14.09 s 2023-09-04 13:16:25,811 44k INFO ====> Epoch: 2654, cost 14.02 s 2023-09-04 13:16:39,856 44k INFO ====> Epoch: 2655, cost 14.05 s 2023-09-04 13:16:54,157 44k INFO ====> Epoch: 2656, cost 14.30 s 2023-09-04 13:17:08,276 44k INFO ====> Epoch: 2657, cost 14.12 s 2023-09-04 13:17:10,333 44k INFO Train Epoch: 2658 [10%] 2023-09-04 13:17:10,333 44k INFO Losses: [2.3121731281280518, 2.3758676052093506, 7.901889801025391, 15.943085670471191, 0.6384047865867615], step: 55800, lr: 7.173827498518913e-05, reference_loss: 29.17142105102539 2023-09-04 13:17:22,643 44k INFO ====> Epoch: 2658, cost 14.37 s 2023-09-04 13:17:36,622 44k INFO ====> Epoch: 2659, cost 13.98 s 2023-09-04 13:17:50,997 44k INFO ====> Epoch: 2660, cost 14.37 s 2023-09-04 13:18:05,297 44k INFO ====> Epoch: 2661, cost 14.30 s 2023-09-04 13:18:19,661 44k INFO ====> Epoch: 2662, cost 14.36 s 2023-09-04 13:18:33,845 44k INFO ====> Epoch: 2663, cost 14.18 s 2023-09-04 13:18:48,051 44k INFO ====> Epoch: 2664, cost 14.21 s 2023-09-04 13:19:02,382 44k INFO ====> Epoch: 2665, cost 14.33 s 2023-09-04 13:19:16,426 44k INFO ====> Epoch: 2666, cost 14.04 s 2023-09-04 13:19:25,947 44k INFO Train Epoch: 2667 [62%] 2023-09-04 13:19:25,948 44k INFO Losses: [2.3988332748413086, 2.566582202911377, 6.656453609466553, 15.323492050170898, 0.6138985753059387], step: 56000, lr: 7.165760976684309e-05, reference_loss: 27.559261322021484 2023-09-04 13:19:30,609 44k INFO Saving model and optimizer state at iteration 2667 to ./logs\44k\G_56000.pth 2023-09-04 13:19:31,239 44k INFO Saving model and optimizer state at iteration 2667 to ./logs\44k\D_56000.pth 2023-09-04 13:19:32,312 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_53600.pth 2023-09-04 13:19:32,347 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_53600.pth 2023-09-04 13:19:36,813 44k INFO ====> Epoch: 2667, cost 20.39 s 2023-09-04 13:19:51,065 44k INFO ====> Epoch: 2668, cost 14.25 s 2023-09-04 13:20:05,338 44k INFO ====> Epoch: 2669, cost 14.27 s 2023-09-04 13:20:19,487 44k INFO ====> Epoch: 2670, cost 14.15 s 2023-09-04 13:20:33,671 44k INFO ====> Epoch: 2671, cost 14.18 s 2023-09-04 13:20:47,908 44k INFO ====> Epoch: 2672, cost 14.24 s 2023-09-04 13:21:02,288 44k INFO ====> Epoch: 2673, cost 14.38 s 2023-09-04 13:21:16,518 44k INFO ====> Epoch: 2674, cost 14.23 s 2023-09-04 13:21:30,492 44k INFO ====> Epoch: 2675, cost 13.97 s 2023-09-04 13:21:44,670 44k INFO ====> Epoch: 2676, cost 14.18 s 2023-09-04 13:21:47,397 44k INFO Train Epoch: 2677 [14%] 2023-09-04 13:21:47,398 44k INFO Losses: [2.3160905838012695, 2.674469232559204, 8.5245361328125, 17.505006790161133, 0.7591219544410706], step: 56200, lr: 7.15680881221003e-05, reference_loss: 31.779226303100586 2023-09-04 13:21:58,944 44k INFO ====> Epoch: 2677, cost 14.27 s 2023-09-04 13:22:13,221 44k INFO ====> Epoch: 2678, cost 14.28 s 2023-09-04 13:22:27,300 44k INFO ====> Epoch: 2679, cost 14.08 s 2023-09-04 13:22:41,419 44k INFO ====> Epoch: 2680, cost 14.12 s 2023-09-04 13:22:55,573 44k INFO ====> Epoch: 2681, cost 14.15 s 2023-09-04 13:23:09,666 44k INFO ====> Epoch: 2682, cost 14.09 s 2023-09-04 13:23:23,620 44k INFO ====> Epoch: 2683, cost 13.95 s 2023-09-04 13:23:37,541 44k INFO ====> Epoch: 2684, cost 13.92 s 2023-09-04 13:23:51,933 44k INFO ====> Epoch: 2685, cost 14.39 s 2023-09-04 13:24:02,386 44k INFO Train Epoch: 2686 [67%] 2023-09-04 13:24:02,387 44k INFO Losses: [2.3388113975524902, 2.5536646842956543, 7.8777055740356445, 16.53366470336914, 0.5667844414710999], step: 56400, lr: 7.148761426827304e-05, reference_loss: 29.870630264282227 2023-09-04 13:24:06,614 44k INFO ====> Epoch: 2686, cost 14.68 s 2023-09-04 13:24:20,943 44k INFO ====> Epoch: 2687, cost 14.33 s 2023-09-04 13:24:35,062 44k INFO ====> Epoch: 2688, cost 14.12 s 2023-09-04 13:24:49,333 44k INFO ====> Epoch: 2689, cost 14.27 s 2023-09-04 13:25:03,618 44k INFO ====> Epoch: 2690, cost 14.29 s 2023-09-04 13:25:17,549 44k INFO ====> Epoch: 2691, cost 13.93 s 2023-09-04 13:25:31,611 44k INFO ====> Epoch: 2692, cost 14.06 s 2023-09-04 13:25:45,680 44k INFO ====> Epoch: 2693, cost 14.07 s 2023-09-04 13:25:59,985 44k INFO ====> Epoch: 2694, cost 14.30 s 2023-09-04 13:26:14,219 44k INFO ====> Epoch: 2695, cost 14.23 s 2023-09-04 13:26:17,640 44k INFO Train Epoch: 2696 [19%] 2023-09-04 13:26:17,640 44k INFO Losses: [2.340153217315674, 2.4360010623931885, 7.5865631103515625, 15.498345375061035, 0.5493663549423218], step: 56600, lr: 7.139830499841519e-05, reference_loss: 28.410429000854492 2023-09-04 13:26:28,474 44k INFO ====> Epoch: 2696, cost 14.25 s 2023-09-04 13:26:42,642 44k INFO ====> Epoch: 2697, cost 14.17 s 2023-09-04 13:26:56,842 44k INFO ====> Epoch: 2698, cost 14.20 s 2023-09-04 13:27:11,156 44k INFO ====> Epoch: 2699, cost 14.31 s 2023-09-04 13:27:25,223 44k INFO ====> Epoch: 2700, cost 14.07 s 2023-09-04 13:27:39,221 44k INFO ====> Epoch: 2701, cost 14.00 s 2023-09-04 13:27:53,456 44k INFO ====> Epoch: 2702, cost 14.24 s 2023-09-04 13:28:07,775 44k INFO ====> Epoch: 2703, cost 14.32 s 2023-09-04 13:28:21,889 44k INFO ====> Epoch: 2704, cost 14.11 s 2023-09-04 13:28:32,745 44k INFO Train Epoch: 2705 [71%] 2023-09-04 13:28:32,745 44k INFO Losses: [2.466082811355591, 2.2957544326782227, 6.741488456726074, 15.17238712310791, 0.6445766687393188], step: 56800, lr: 7.131802205512693e-05, reference_loss: 27.320289611816406 2023-09-04 13:28:37,498 44k INFO Saving model and optimizer state at iteration 2705 to ./logs\44k\G_56800.pth 2023-09-04 13:28:38,090 44k INFO Saving model and optimizer state at iteration 2705 to ./logs\44k\D_56800.pth 2023-09-04 13:28:39,305 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_54400.pth 2023-09-04 13:28:39,340 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_54400.pth 2023-09-04 13:28:42,588 44k INFO ====> Epoch: 2705, cost 20.70 s 2023-09-04 13:28:56,847 44k INFO ====> Epoch: 2706, cost 14.26 s 2023-09-04 13:29:11,088 44k INFO ====> Epoch: 2707, cost 14.24 s 2023-09-04 13:29:25,070 44k INFO ====> Epoch: 2708, cost 13.98 s 2023-09-04 13:29:39,060 44k INFO ====> Epoch: 2709, cost 13.99 s 2023-09-04 13:29:53,508 44k INFO ====> Epoch: 2710, cost 14.45 s 2023-09-04 13:30:07,881 44k INFO ====> Epoch: 2711, cost 14.37 s 2023-09-04 13:30:22,014 44k INFO ====> Epoch: 2712, cost 14.13 s 2023-09-04 13:30:36,144 44k INFO ====> Epoch: 2713, cost 14.13 s 2023-09-04 13:30:50,493 44k INFO ====> Epoch: 2714, cost 14.35 s 2023-09-04 13:30:54,644 44k INFO Train Epoch: 2715 [24%] 2023-09-04 13:30:54,645 44k INFO Losses: [2.333118438720703, 2.4453847408294678, 7.260828971862793, 15.039064407348633, 0.6441304683685303], step: 57000, lr: 7.122892465633075e-05, reference_loss: 27.72252655029297 2023-09-04 13:31:05,055 44k INFO ====> Epoch: 2715, cost 14.56 s 2023-09-04 13:31:19,080 44k INFO ====> Epoch: 2716, cost 14.03 s 2023-09-04 13:31:33,142 44k INFO ====> Epoch: 2717, cost 14.06 s 2023-09-04 13:31:47,311 44k INFO ====> Epoch: 2718, cost 14.17 s 2023-09-04 13:32:01,463 44k INFO ====> Epoch: 2719, cost 14.15 s 2023-09-04 13:32:15,709 44k INFO ====> Epoch: 2720, cost 14.25 s 2023-09-04 13:32:29,839 44k INFO ====> Epoch: 2721, cost 14.13 s 2023-09-04 13:32:44,031 44k INFO ====> Epoch: 2722, cost 14.19 s 2023-09-04 13:32:58,202 44k INFO ====> Epoch: 2723, cost 14.17 s 2023-09-04 13:33:09,833 44k INFO Train Epoch: 2724 [76%] 2023-09-04 13:33:09,833 44k INFO Losses: [2.286978006362915, 2.553398609161377, 8.579233169555664, 17.5301570892334, 0.5724316835403442], step: 57200, lr: 7.114883217067864e-05, reference_loss: 31.522199630737305 2023-09-04 13:33:12,551 44k INFO ====> Epoch: 2724, cost 14.35 s 2023-09-04 13:33:26,540 44k INFO ====> Epoch: 2725, cost 13.99 s 2023-09-04 13:33:40,630 44k INFO ====> Epoch: 2726, cost 14.09 s 2023-09-04 13:33:54,712 44k INFO ====> Epoch: 2727, cost 14.08 s 2023-09-04 13:34:08,887 44k INFO ====> Epoch: 2728, cost 14.17 s 2023-09-04 13:34:22,985 44k INFO ====> Epoch: 2729, cost 14.10 s 2023-09-04 13:34:37,017 44k INFO ====> Epoch: 2730, cost 14.03 s 2023-09-04 13:34:51,162 44k INFO ====> Epoch: 2731, cost 14.14 s 2023-09-04 13:35:05,345 44k INFO ====> Epoch: 2732, cost 14.18 s 2023-09-04 13:35:19,317 44k INFO ====> Epoch: 2733, cost 13.97 s 2023-09-04 13:35:24,096 44k INFO Train Epoch: 2734 [29%] 2023-09-04 13:35:24,097 44k INFO Losses: [2.266993284225464, 2.465773582458496, 6.9929094314575195, 14.34438419342041, 0.4942706525325775], step: 57400, lr: 7.105994614031605e-05, reference_loss: 26.5643310546875 2023-09-04 13:35:33,613 44k INFO ====> Epoch: 2734, cost 14.30 s 2023-09-04 13:35:47,829 44k INFO ====> Epoch: 2735, cost 14.22 s 2023-09-04 13:36:02,229 44k INFO ====> Epoch: 2736, cost 14.40 s 2023-09-04 13:36:16,557 44k INFO ====> Epoch: 2737, cost 14.33 s 2023-09-04 13:36:30,725 44k INFO ====> Epoch: 2738, cost 14.17 s 2023-09-04 13:36:44,947 44k INFO ====> Epoch: 2739, cost 14.22 s 2023-09-04 13:36:59,254 44k INFO ====> Epoch: 2740, cost 14.31 s 2023-09-04 13:37:13,539 44k INFO ====> Epoch: 2741, cost 14.29 s 2023-09-04 13:37:27,530 44k INFO ====> Epoch: 2742, cost 13.99 s 2023-09-04 13:37:39,721 44k INFO Train Epoch: 2743 [81%] 2023-09-04 13:37:39,721 44k INFO Losses: [2.3039581775665283, 2.740532398223877, 7.106482982635498, 15.40744686126709, 0.6364243626594543], step: 57600, lr: 7.09800436604718e-05, reference_loss: 28.194847106933594 2023-09-04 13:37:44,474 44k INFO Saving model and optimizer state at iteration 2743 to ./logs\44k\G_57600.pth 2023-09-04 13:37:45,125 44k INFO Saving model and optimizer state at iteration 2743 to ./logs\44k\D_57600.pth 2023-09-04 13:37:46,704 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_55200.pth 2023-09-04 13:37:46,735 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_55200.pth 2023-09-04 13:37:48,533 44k INFO ====> Epoch: 2743, cost 21.00 s 2023-09-04 13:38:02,860 44k INFO ====> Epoch: 2744, cost 14.33 s 2023-09-04 13:38:17,100 44k INFO ====> Epoch: 2745, cost 14.24 s 2023-09-04 13:38:31,006 44k INFO ====> Epoch: 2746, cost 13.91 s 2023-09-04 13:38:45,233 44k INFO ====> Epoch: 2747, cost 14.23 s 2023-09-04 13:38:59,517 44k INFO ====> Epoch: 2748, cost 14.28 s 2023-09-04 13:39:13,564 44k INFO ====> Epoch: 2749, cost 14.05 s 2023-09-04 13:39:27,656 44k INFO ====> Epoch: 2750, cost 14.09 s 2023-09-04 13:39:41,621 44k INFO ====> Epoch: 2751, cost 13.97 s 2023-09-04 13:39:55,945 44k INFO ====> Epoch: 2752, cost 14.32 s 2023-09-04 13:40:01,404 44k INFO Train Epoch: 2753 [33%] 2023-09-04 13:40:01,404 44k INFO Losses: [2.2394490242004395, 2.8949382305145264, 8.414091110229492, 15.644468307495117, 0.6320595741271973], step: 57800, lr: 7.089136849710706e-05, reference_loss: 29.82500648498535 2023-09-04 13:40:10,346 44k INFO ====> Epoch: 2753, cost 14.40 s 2023-09-04 13:40:24,319 44k INFO ====> Epoch: 2754, cost 13.97 s 2023-09-04 13:40:38,220 44k INFO ====> Epoch: 2755, cost 13.90 s 2023-09-04 13:40:52,576 44k INFO ====> Epoch: 2756, cost 14.36 s 2023-09-04 13:41:07,001 44k INFO ====> Epoch: 2757, cost 14.43 s 2023-09-04 13:41:21,084 44k INFO ====> Epoch: 2758, cost 14.08 s 2023-09-04 13:41:35,047 44k INFO ====> Epoch: 2759, cost 13.96 s 2023-09-04 13:41:49,221 44k INFO ====> Epoch: 2760, cost 14.17 s 2023-09-04 13:42:03,738 44k INFO ====> Epoch: 2761, cost 14.52 s 2023-09-04 13:42:16,814 44k INFO Train Epoch: 2762 [86%] 2023-09-04 13:42:16,814 44k INFO Losses: [2.376943349838257, 2.545194149017334, 7.275724411010742, 15.691951751708984, 0.5576218366622925], step: 58000, lr: 7.081165557231414e-05, reference_loss: 28.447433471679688 2023-09-04 13:42:18,183 44k INFO ====> Epoch: 2762, cost 14.45 s 2023-09-04 13:42:32,248 44k INFO ====> Epoch: 2763, cost 14.06 s 2023-09-04 13:42:46,405 44k INFO ====> Epoch: 2764, cost 14.16 s 2023-09-04 13:43:00,815 44k INFO ====> Epoch: 2765, cost 14.41 s 2023-09-04 13:43:14,958 44k INFO ====> Epoch: 2766, cost 14.14 s 2023-09-04 13:43:28,883 44k INFO ====> Epoch: 2767, cost 13.92 s 2023-09-04 13:43:43,020 44k INFO ====> Epoch: 2768, cost 14.14 s 2023-09-04 13:43:57,360 44k INFO ====> Epoch: 2769, cost 14.34 s 2023-09-04 13:44:11,515 44k INFO ====> Epoch: 2770, cost 14.16 s 2023-09-04 13:44:25,720 44k INFO ====> Epoch: 2771, cost 14.21 s 2023-09-04 13:44:31,908 44k INFO Train Epoch: 2772 [38%] 2023-09-04 13:44:31,908 44k INFO Losses: [2.6291391849517822, 2.6349565982818604, 6.539022445678711, 16.10479736328125, 0.48281195759773254], step: 58200, lr: 7.07231907757012e-05, reference_loss: 28.390727996826172 2023-09-04 13:44:40,065 44k INFO ====> Epoch: 2772, cost 14.34 s 2023-09-04 13:44:54,227 44k INFO ====> Epoch: 2773, cost 14.16 s 2023-09-04 13:45:08,586 44k INFO ====> Epoch: 2774, cost 14.36 s 2023-09-04 13:45:22,729 44k INFO ====> Epoch: 2775, cost 14.14 s 2023-09-04 13:45:36,692 44k INFO ====> Epoch: 2776, cost 13.96 s 2023-09-04 13:45:50,728 44k INFO ====> Epoch: 2777, cost 14.04 s 2023-09-04 13:46:05,018 44k INFO ====> Epoch: 2778, cost 14.29 s 2023-09-04 13:46:19,126 44k INFO ====> Epoch: 2779, cost 14.11 s 2023-09-04 13:46:33,177 44k INFO ====> Epoch: 2780, cost 14.05 s 2023-09-04 13:46:46,844 44k INFO Train Epoch: 2781 [90%] 2023-09-04 13:46:46,844 44k INFO Losses: [2.364086389541626, 2.5274267196655273, 7.87099027633667, 17.11836814880371, 0.7273091077804565], step: 58400, lr: 7.064366695627249e-05, reference_loss: 30.608179092407227 2023-09-04 13:46:51,615 44k INFO Saving model and optimizer state at iteration 2781 to ./logs\44k\G_58400.pth 2023-09-04 13:46:52,227 44k INFO Saving model and optimizer state at iteration 2781 to ./logs\44k\D_58400.pth 2023-09-04 13:46:53,453 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_56000.pth 2023-09-04 13:46:53,487 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_56000.pth 2023-09-04 13:46:53,930 44k INFO ====> Epoch: 2781, cost 20.75 s 2023-09-04 13:47:08,232 44k INFO ====> Epoch: 2782, cost 14.30 s 2023-09-04 13:47:22,272 44k INFO ====> Epoch: 2783, cost 14.04 s 2023-09-04 13:47:36,354 44k INFO ====> Epoch: 2784, cost 14.08 s 2023-09-04 13:47:50,672 44k INFO ====> Epoch: 2785, cost 14.32 s 2023-09-04 13:48:05,091 44k INFO ====> Epoch: 2786, cost 14.42 s 2023-09-04 13:48:19,358 44k INFO ====> Epoch: 2787, cost 14.27 s 2023-09-04 13:48:33,394 44k INFO ====> Epoch: 2788, cost 14.04 s 2023-09-04 13:48:47,627 44k INFO ====> Epoch: 2789, cost 14.23 s 2023-09-04 13:49:01,892 44k INFO ====> Epoch: 2790, cost 14.26 s 2023-09-04 13:49:08,806 44k INFO Train Epoch: 2791 [43%] 2023-09-04 13:49:08,806 44k INFO Losses: [2.3070390224456787, 2.3928370475769043, 8.004582405090332, 16.574466705322266, 0.61894291639328], step: 58600, lr: 7.055541202735195e-05, reference_loss: 29.89786720275879 2023-09-04 13:49:16,308 44k INFO ====> Epoch: 2791, cost 14.42 s 2023-09-04 13:49:30,179 44k INFO ====> Epoch: 2792, cost 13.87 s 2023-09-04 13:49:44,539 44k INFO ====> Epoch: 2793, cost 14.36 s 2023-09-04 13:49:58,723 44k INFO ====> Epoch: 2794, cost 14.18 s 2023-09-04 13:50:12,914 44k INFO ====> Epoch: 2795, cost 14.19 s 2023-09-04 13:50:27,132 44k INFO ====> Epoch: 2796, cost 14.22 s 2023-09-04 13:50:41,313 44k INFO ====> Epoch: 2797, cost 14.18 s 2023-09-04 13:50:55,572 44k INFO ====> Epoch: 2798, cost 14.26 s 2023-09-04 13:51:09,818 44k INFO ====> Epoch: 2799, cost 14.25 s 2023-09-04 13:51:23,744 44k INFO Train Epoch: 2800 [95%] 2023-09-04 13:51:23,745 44k INFO Losses: [2.2363317012786865, 3.049480438232422, 9.37814712524414, 16.87612533569336, 0.23537202179431915], step: 58800, lr: 7.04760768646671e-05, reference_loss: 31.77545738220215 2023-09-04 13:51:23,989 44k INFO ====> Epoch: 2800, cost 14.17 s 2023-09-04 13:51:38,060 44k INFO ====> Epoch: 2801, cost 14.07 s 2023-09-04 13:51:52,056 44k INFO ====> Epoch: 2802, cost 14.00 s 2023-09-04 13:52:06,093 44k INFO ====> Epoch: 2803, cost 14.04 s 2023-09-04 13:52:20,213 44k INFO ====> Epoch: 2804, cost 14.12 s 2023-09-04 13:52:34,262 44k INFO ====> Epoch: 2805, cost 14.05 s 2023-09-04 13:52:48,505 44k INFO ====> Epoch: 2806, cost 14.24 s 2023-09-04 13:53:02,662 44k INFO ====> Epoch: 2807, cost 14.16 s 2023-09-04 13:53:16,749 44k INFO ====> Epoch: 2808, cost 14.09 s 2023-09-04 13:53:30,584 44k INFO ====> Epoch: 2809, cost 13.84 s 2023-09-04 13:53:38,031 44k INFO Train Epoch: 2810 [48%] 2023-09-04 13:53:38,031 44k INFO Losses: [2.165938377380371, 3.090548515319824, 8.280223846435547, 16.13990020751953, 0.6009241938591003], step: 59000, lr: 7.038803130556357e-05, reference_loss: 30.27753448486328 2023-09-04 13:53:44,936 44k INFO ====> Epoch: 2810, cost 14.35 s 2023-09-04 13:53:59,214 44k INFO ====> Epoch: 2811, cost 14.28 s 2023-09-04 13:54:13,482 44k INFO ====> Epoch: 2812, cost 14.27 s 2023-09-04 13:54:27,653 44k INFO ====> Epoch: 2813, cost 14.17 s 2023-09-04 13:54:41,715 44k INFO ====> Epoch: 2814, cost 14.06 s 2023-09-04 13:54:56,065 44k INFO ====> Epoch: 2815, cost 14.35 s 2023-09-04 13:55:10,435 44k INFO ====> Epoch: 2816, cost 14.37 s 2023-09-04 13:55:24,364 44k INFO ====> Epoch: 2817, cost 13.93 s 2023-09-04 13:55:38,379 44k INFO ====> Epoch: 2818, cost 14.02 s 2023-09-04 13:55:52,639 44k INFO ====> Epoch: 2819, cost 14.26 s 2023-09-04 13:55:53,370 44k INFO Train Epoch: 2820 [0%] 2023-09-04 13:55:53,370 44k INFO Losses: [2.5618093013763428, 2.380181312561035, 6.87270975112915, 15.011449813842773, 0.6683027148246765], step: 59200, lr: 7.030009574152251e-05, reference_loss: 27.49445343017578 2023-09-04 13:55:58,008 44k INFO Saving model and optimizer state at iteration 2820 to ./logs\44k\G_59200.pth 2023-09-04 13:55:58,587 44k INFO Saving model and optimizer state at iteration 2820 to ./logs\44k\D_59200.pth 2023-09-04 13:56:00,161 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_56800.pth 2023-09-04 13:56:00,196 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_56800.pth 2023-09-04 13:56:13,724 44k INFO ====> Epoch: 2820, cost 21.08 s 2023-09-04 13:56:27,863 44k INFO ====> Epoch: 2821, cost 14.14 s 2023-09-04 13:56:42,029 44k INFO ====> Epoch: 2822, cost 14.17 s 2023-09-04 13:56:56,290 44k INFO ====> Epoch: 2823, cost 14.26 s 2023-09-04 13:57:10,500 44k INFO ====> Epoch: 2824, cost 14.21 s 2023-09-04 13:57:24,593 44k INFO ====> Epoch: 2825, cost 14.09 s 2023-09-04 13:57:38,609 44k INFO ====> Epoch: 2826, cost 14.02 s 2023-09-04 13:57:52,709 44k INFO ====> Epoch: 2827, cost 14.10 s 2023-09-04 13:58:06,708 44k INFO ====> Epoch: 2828, cost 14.00 s 2023-09-04 13:58:14,884 44k INFO Train Epoch: 2829 [52%] 2023-09-04 13:58:14,884 44k INFO Losses: [2.3215441703796387, 2.588794469833374, 7.2835516929626465, 15.223098754882812, 0.698082685470581], step: 59400, lr: 7.022104766608568e-05, reference_loss: 28.115070343017578 2023-09-04 13:58:21,077 44k INFO ====> Epoch: 2829, cost 14.37 s 2023-09-04 13:58:35,138 44k INFO ====> Epoch: 2830, cost 14.06 s 2023-09-04 13:58:49,304 44k INFO ====> Epoch: 2831, cost 14.17 s 2023-09-04 13:59:03,389 44k INFO ====> Epoch: 2832, cost 14.09 s 2023-09-04 13:59:17,673 44k INFO ====> Epoch: 2833, cost 14.28 s 2023-09-04 13:59:31,634 44k INFO ====> Epoch: 2834, cost 13.96 s 2023-09-04 13:59:46,019 44k INFO ====> Epoch: 2835, cost 14.38 s 2023-09-04 14:00:00,475 44k INFO ====> Epoch: 2836, cost 14.46 s 2023-09-04 14:00:14,717 44k INFO ====> Epoch: 2837, cost 14.24 s 2023-09-04 14:00:28,787 44k INFO ====> Epoch: 2838, cost 14.07 s 2023-09-04 14:00:30,140 44k INFO Train Epoch: 2839 [5%] 2023-09-04 14:00:30,140 44k INFO Losses: [2.485537528991699, 2.30015230178833, 8.18125057220459, 14.894859313964844, 0.546285092830658], step: 59600, lr: 7.013332071422273e-05, reference_loss: 28.408084869384766 2023-09-04 14:00:43,238 44k INFO ====> Epoch: 2839, cost 14.45 s 2023-09-04 14:00:57,546 44k INFO ====> Epoch: 2840, cost 14.31 s 2023-09-04 14:01:11,957 44k INFO ====> Epoch: 2841, cost 14.41 s 2023-09-04 14:01:25,831 44k INFO ====> Epoch: 2842, cost 13.87 s 2023-09-04 14:01:39,918 44k INFO ====> Epoch: 2843, cost 14.09 s 2023-09-04 14:01:54,325 44k INFO ====> Epoch: 2844, cost 14.41 s 2023-09-04 14:02:08,701 44k INFO ====> Epoch: 2845, cost 14.38 s 2023-09-04 14:02:22,785 44k INFO ====> Epoch: 2846, cost 14.08 s 2023-09-04 14:02:36,838 44k INFO ====> Epoch: 2847, cost 14.05 s 2023-09-04 14:02:45,684 44k INFO Train Epoch: 2848 [57%] 2023-09-04 14:02:45,685 44k INFO Losses: [2.269374370574951, 2.58817720413208, 8.711620330810547, 16.849916458129883, 0.5088031888008118], step: 59800, lr: 7.0054460166908e-05, reference_loss: 30.92789077758789 2023-09-04 14:02:51,236 44k INFO ====> Epoch: 2848, cost 14.40 s 2023-09-04 14:03:05,453 44k INFO ====> Epoch: 2849, cost 14.22 s 2023-09-04 14:03:19,541 44k INFO ====> Epoch: 2850, cost 14.09 s 2023-09-04 14:03:33,580 44k INFO ====> Epoch: 2851, cost 14.04 s 2023-09-04 14:03:47,799 44k INFO ====> Epoch: 2852, cost 14.22 s 2023-09-04 14:04:01,754 44k INFO ====> Epoch: 2853, cost 13.96 s 2023-09-04 14:04:15,902 44k INFO ====> Epoch: 2854, cost 14.15 s 2023-09-04 14:04:30,102 44k INFO ====> Epoch: 2855, cost 14.20 s 2023-09-04 14:04:44,166 44k INFO ====> Epoch: 2856, cost 14.06 s 2023-09-04 14:04:58,196 44k INFO ====> Epoch: 2857, cost 14.03 s 2023-09-04 14:05:00,286 44k INFO Train Epoch: 2858 [10%] 2023-09-04 14:05:00,286 44k INFO Losses: [2.5164413452148438, 2.1323397159576416, 6.5650105476379395, 15.289721488952637, 0.7740009427070618], step: 60000, lr: 6.996694133232622e-05, reference_loss: 27.27751350402832 2023-09-04 14:05:04,972 44k INFO Saving model and optimizer state at iteration 2858 to ./logs\44k\G_60000.pth 2023-09-04 14:05:05,566 44k INFO Saving model and optimizer state at iteration 2858 to ./logs\44k\D_60000.pth 2023-09-04 14:05:06,839 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_57600.pth 2023-09-04 14:05:06,873 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_57600.pth 2023-09-04 14:05:19,059 44k INFO ====> Epoch: 2858, cost 20.86 s 2023-09-04 14:05:33,002 44k INFO ====> Epoch: 2859, cost 13.94 s 2023-09-04 14:05:47,079 44k INFO ====> Epoch: 2860, cost 14.08 s 2023-09-04 14:06:01,538 44k INFO ====> Epoch: 2861, cost 14.46 s 2023-09-04 14:06:15,832 44k INFO ====> Epoch: 2862, cost 14.29 s 2023-09-04 14:06:29,984 44k INFO ====> Epoch: 2863, cost 14.15 s 2023-09-04 14:06:44,238 44k INFO ====> Epoch: 2864, cost 14.25 s 2023-09-04 14:06:58,594 44k INFO ====> Epoch: 2865, cost 14.36 s 2023-09-04 14:07:12,804 44k INFO ====> Epoch: 2866, cost 14.21 s 2023-09-04 14:07:22,318 44k INFO Train Epoch: 2867 [62%] 2023-09-04 14:07:22,318 44k INFO Losses: [2.205413341522217, 2.836526393890381, 8.769487380981445, 17.07813835144043, 0.5167827606201172], step: 60200, lr: 6.988826786825501e-05, reference_loss: 31.406347274780273 2023-09-04 14:07:27,098 44k INFO ====> Epoch: 2867, cost 14.29 s 2023-09-04 14:07:41,293 44k INFO ====> Epoch: 2868, cost 14.20 s 2023-09-04 14:07:55,628 44k INFO ====> Epoch: 2869, cost 14.33 s 2023-09-04 14:08:09,995 44k INFO ====> Epoch: 2870, cost 14.37 s 2023-09-04 14:08:24,051 44k INFO ====> Epoch: 2871, cost 14.06 s 2023-09-04 14:08:38,264 44k INFO ====> Epoch: 2872, cost 14.21 s 2023-09-04 14:08:52,524 44k INFO ====> Epoch: 2873, cost 14.26 s 2023-09-04 14:09:06,609 44k INFO ====> Epoch: 2874, cost 14.09 s 2023-09-04 14:09:20,652 44k INFO ====> Epoch: 2875, cost 14.04 s 2023-09-04 14:09:34,549 44k INFO ====> Epoch: 2876, cost 13.90 s 2023-09-04 14:09:37,319 44k INFO Train Epoch: 2877 [14%] 2023-09-04 14:09:37,319 44k INFO Losses: [2.3705461025238037, 2.378113031387329, 7.279240608215332, 16.255870819091797, 0.5556697249412537], step: 60400, lr: 6.980095665723153e-05, reference_loss: 28.839441299438477 2023-09-04 14:09:49,098 44k INFO ====> Epoch: 2877, cost 14.55 s 2023-09-04 14:10:03,300 44k INFO ====> Epoch: 2878, cost 14.20 s 2023-09-04 14:10:17,496 44k INFO ====> Epoch: 2879, cost 14.20 s 2023-09-04 14:10:31,473 44k INFO ====> Epoch: 2880, cost 13.98 s 2023-09-04 14:10:45,621 44k INFO ====> Epoch: 2881, cost 14.15 s 2023-09-04 14:11:00,005 44k INFO ====> Epoch: 2882, cost 14.39 s 2023-09-04 14:11:14,250 44k INFO ====> Epoch: 2883, cost 14.24 s 2023-09-04 14:11:28,429 44k INFO ====> Epoch: 2884, cost 14.18 s 2023-09-04 14:11:42,639 44k INFO ====> Epoch: 2885, cost 14.21 s 2023-09-04 14:11:53,170 44k INFO Train Epoch: 2886 [67%] 2023-09-04 14:11:53,171 44k INFO Losses: [2.245950222015381, 2.5462377071380615, 6.615924835205078, 13.276483535766602, 0.5033122301101685], step: 60600, lr: 6.972246983258066e-05, reference_loss: 25.187910079956055 2023-09-04 14:11:57,443 44k INFO ====> Epoch: 2886, cost 14.80 s 2023-09-04 14:12:11,812 44k INFO ====> Epoch: 2887, cost 14.37 s 2023-09-04 14:12:25,927 44k INFO ====> Epoch: 2888, cost 14.11 s 2023-09-04 14:12:40,132 44k INFO ====> Epoch: 2889, cost 14.21 s 2023-09-04 14:12:54,655 44k INFO ====> Epoch: 2890, cost 14.52 s 2023-09-04 14:13:08,970 44k INFO ====> Epoch: 2891, cost 14.31 s 2023-09-04 14:13:23,141 44k INFO ====> Epoch: 2892, cost 14.17 s 2023-09-04 14:13:37,195 44k INFO ====> Epoch: 2893, cost 14.05 s 2023-09-04 14:13:51,429 44k INFO ====> Epoch: 2894, cost 14.23 s 2023-09-04 14:14:05,742 44k INFO ====> Epoch: 2895, cost 14.31 s 2023-09-04 14:14:09,271 44k INFO Train Epoch: 2896 [19%] 2023-09-04 14:14:09,272 44k INFO Losses: [2.4937002658843994, 2.22080135345459, 6.2824201583862305, 15.633129119873047, 0.7148633599281311], step: 60800, lr: 6.963536575256389e-05, reference_loss: 27.344913482666016 2023-09-04 14:14:13,937 44k INFO Saving model and optimizer state at iteration 2896 to ./logs\44k\G_60800.pth 2023-09-04 14:14:14,593 44k INFO Saving model and optimizer state at iteration 2896 to ./logs\44k\D_60800.pth 2023-09-04 14:14:15,826 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_58400.pth 2023-09-04 14:14:15,859 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_58400.pth 2023-09-04 14:14:26,654 44k INFO ====> Epoch: 2896, cost 20.91 s 2023-09-04 14:14:40,588 44k INFO ====> Epoch: 2897, cost 13.93 s 2023-09-04 14:14:54,830 44k INFO ====> Epoch: 2898, cost 14.24 s 2023-09-04 14:15:08,935 44k INFO ====> Epoch: 2899, cost 14.10 s 2023-09-04 14:15:22,939 44k INFO ====> Epoch: 2900, cost 14.00 s 2023-09-04 14:15:36,954 44k INFO ====> Epoch: 2901, cost 14.02 s 2023-09-04 14:15:51,102 44k INFO ====> Epoch: 2902, cost 14.15 s 2023-09-04 14:16:05,239 44k INFO ====> Epoch: 2903, cost 14.14 s 2023-09-04 14:16:19,318 44k INFO ====> Epoch: 2904, cost 14.08 s 2023-09-04 14:16:30,096 44k INFO Train Epoch: 2905 [71%] 2023-09-04 14:16:30,096 44k INFO Losses: [2.32119083404541, 2.4383838176727295, 6.5923051834106445, 14.728328704833984, 0.743549644947052], step: 61000, lr: 6.955706512456305e-05, reference_loss: 26.82375717163086 2023-09-04 14:16:33,499 44k INFO ====> Epoch: 2905, cost 14.18 s 2023-09-04 14:16:47,682 44k INFO ====> Epoch: 2906, cost 14.18 s 2023-09-04 14:17:01,836 44k INFO ====> Epoch: 2907, cost 14.15 s 2023-09-04 14:17:16,010 44k INFO ====> Epoch: 2908, cost 14.17 s 2023-09-04 14:17:29,937 44k INFO ====> Epoch: 2909, cost 13.93 s 2023-09-04 14:17:44,021 44k INFO ====> Epoch: 2910, cost 14.08 s 2023-09-04 14:17:58,366 44k INFO ====> Epoch: 2911, cost 14.34 s 2023-09-04 14:18:12,702 44k INFO ====> Epoch: 2912, cost 14.34 s 2023-09-04 14:18:26,845 44k INFO ====> Epoch: 2913, cost 14.14 s 2023-09-04 14:18:41,038 44k INFO ====> Epoch: 2914, cost 14.19 s 2023-09-04 14:18:45,254 44k INFO Train Epoch: 2915 [24%] 2023-09-04 14:18:45,254 44k INFO Losses: [2.4804818630218506, 2.2735748291015625, 6.454238414764404, 15.141634941101074, 0.5136884450912476], step: 61200, lr: 6.947016768416984e-05, reference_loss: 26.863618850708008 2023-09-04 14:18:55,856 44k INFO ====> Epoch: 2915, cost 14.82 s 2023-09-04 14:19:10,162 44k INFO ====> Epoch: 2916, cost 14.31 s 2023-09-04 14:19:24,068 44k INFO ====> Epoch: 2917, cost 13.91 s 2023-09-04 14:19:38,039 44k INFO ====> Epoch: 2918, cost 13.97 s 2023-09-04 14:19:52,454 44k INFO ====> Epoch: 2919, cost 14.41 s 2023-09-04 14:20:06,638 44k INFO ====> Epoch: 2920, cost 14.18 s 2023-09-04 14:20:20,803 44k INFO ====> Epoch: 2921, cost 14.17 s 2023-09-04 14:20:34,973 44k INFO ====> Epoch: 2922, cost 14.17 s 2023-09-04 14:20:49,163 44k INFO ====> Epoch: 2923, cost 14.19 s 2023-09-04 14:21:01,072 44k INFO Train Epoch: 2924 [76%] 2023-09-04 14:21:01,072 44k INFO Losses: [2.593143939971924, 2.1978235244750977, 6.226171970367432, 13.963751792907715, 0.4713803827762604], step: 61400, lr: 6.939205281109914e-05, reference_loss: 25.452272415161133 2023-09-04 14:21:03,777 44k INFO ====> Epoch: 2924, cost 14.61 s 2023-09-04 14:21:17,809 44k INFO ====> Epoch: 2925, cost 14.03 s 2023-09-04 14:21:31,826 44k INFO ====> Epoch: 2926, cost 14.02 s 2023-09-04 14:21:46,052 44k INFO ====> Epoch: 2927, cost 14.23 s 2023-09-04 14:22:00,287 44k INFO ====> Epoch: 2928, cost 14.23 s 2023-09-04 14:22:14,576 44k INFO ====> Epoch: 2929, cost 14.29 s 2023-09-04 14:22:28,628 44k INFO ====> Epoch: 2930, cost 14.05 s 2023-09-04 14:22:42,840 44k INFO ====> Epoch: 2931, cost 14.21 s 2023-09-04 14:22:57,032 44k INFO ====> Epoch: 2932, cost 14.19 s 2023-09-04 14:23:11,236 44k INFO ====> Epoch: 2933, cost 14.20 s 2023-09-04 14:23:15,972 44k INFO Train Epoch: 2934 [29%] 2023-09-04 14:23:15,972 44k INFO Losses: [2.283169746398926, 2.5254108905792236, 9.360932350158691, 18.3056640625, 0.7294973731040955], step: 61600, lr: 6.930536152011217e-05, reference_loss: 33.204673767089844 2023-09-04 14:23:20,651 44k INFO Saving model and optimizer state at iteration 2934 to ./logs\44k\G_61600.pth 2023-09-04 14:23:21,224 44k INFO Saving model and optimizer state at iteration 2934 to ./logs\44k\D_61600.pth 2023-09-04 14:23:22,453 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_59200.pth 2023-09-04 14:23:22,490 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_59200.pth 2023-09-04 14:23:31,866 44k INFO ====> Epoch: 2934, cost 20.63 s 2023-09-04 14:23:45,997 44k INFO ====> Epoch: 2935, cost 14.13 s 2023-09-04 14:24:00,364 44k INFO ====> Epoch: 2936, cost 14.37 s 2023-09-04 14:24:14,728 44k INFO ====> Epoch: 2937, cost 14.36 s 2023-09-04 14:24:28,773 44k INFO ====> Epoch: 2938, cost 14.04 s 2023-09-04 14:24:42,806 44k INFO ====> Epoch: 2939, cost 14.03 s 2023-09-04 14:24:57,068 44k INFO ====> Epoch: 2940, cost 14.26 s 2023-09-04 14:25:11,306 44k INFO ====> Epoch: 2941, cost 14.24 s 2023-09-04 14:25:25,279 44k INFO ====> Epoch: 2942, cost 13.97 s 2023-09-04 14:25:37,408 44k INFO Train Epoch: 2943 [81%] 2023-09-04 14:25:37,408 44k INFO Losses: [2.2853176593780518, 2.5947108268737793, 7.5212721824646, 16.3072566986084, 0.6627618074417114], step: 61800, lr: 6.922743196129958e-05, reference_loss: 29.371318817138672 2023-09-04 14:25:39,437 44k INFO ====> Epoch: 2943, cost 14.16 s 2023-09-04 14:25:53,669 44k INFO ====> Epoch: 2944, cost 14.23 s 2023-09-04 14:26:07,946 44k INFO ====> Epoch: 2945, cost 14.28 s 2023-09-04 14:26:22,093 44k INFO ====> Epoch: 2946, cost 14.15 s 2023-09-04 14:26:36,341 44k INFO ====> Epoch: 2947, cost 14.25 s 2023-09-04 14:26:50,382 44k INFO ====> Epoch: 2948, cost 14.04 s 2023-09-04 14:27:04,473 44k INFO ====> Epoch: 2949, cost 14.09 s 2023-09-04 14:27:18,391 44k INFO ====> Epoch: 2950, cost 13.92 s 2023-09-04 14:27:32,341 44k INFO ====> Epoch: 2951, cost 13.95 s 2023-09-04 14:27:46,341 44k INFO ====> Epoch: 2952, cost 14.00 s 2023-09-04 14:27:51,762 44k INFO Train Epoch: 2953 [33%] 2023-09-04 14:27:51,763 44k INFO Losses: [2.3463878631591797, 2.6512041091918945, 8.21544361114502, 16.361146926879883, 0.6393049359321594], step: 62000, lr: 6.914094633066439e-05, reference_loss: 30.21348762512207 2023-09-04 14:28:00,817 44k INFO ====> Epoch: 2953, cost 14.48 s 2023-09-04 14:28:14,851 44k INFO ====> Epoch: 2954, cost 14.03 s 2023-09-04 14:28:28,803 44k INFO ====> Epoch: 2955, cost 13.95 s 2023-09-04 14:28:42,812 44k INFO ====> Epoch: 2956, cost 14.01 s 2023-09-04 14:28:57,117 44k INFO ====> Epoch: 2957, cost 14.31 s 2023-09-04 14:29:11,418 44k INFO ====> Epoch: 2958, cost 14.30 s 2023-09-04 14:29:25,520 44k INFO ====> Epoch: 2959, cost 14.10 s 2023-09-04 14:29:39,550 44k INFO ====> Epoch: 2960, cost 14.03 s 2023-09-04 14:29:53,888 44k INFO ====> Epoch: 2961, cost 14.34 s 2023-09-04 14:30:07,354 44k INFO Train Epoch: 2962 [86%] 2023-09-04 14:30:07,355 44k INFO Losses: [2.4526405334472656, 2.3907113075256348, 6.794349193572998, 15.38570499420166, 0.6970213651657104], step: 62200, lr: 6.906320164648336e-05, reference_loss: 27.720428466796875 2023-09-04 14:30:08,730 44k INFO ====> Epoch: 2962, cost 14.84 s 2023-09-04 14:30:22,930 44k INFO ====> Epoch: 2963, cost 14.20 s 2023-09-04 14:30:37,103 44k INFO ====> Epoch: 2964, cost 14.17 s 2023-09-04 14:30:51,263 44k INFO ====> Epoch: 2965, cost 14.16 s 2023-09-04 14:31:05,593 44k INFO ====> Epoch: 2966, cost 14.33 s 2023-09-04 14:31:19,741 44k INFO ====> Epoch: 2967, cost 14.15 s 2023-09-04 14:31:33,680 44k INFO ====> Epoch: 2968, cost 13.94 s 2023-09-04 14:31:47,938 44k INFO ====> Epoch: 2969, cost 14.26 s 2023-09-04 14:32:02,285 44k INFO ====> Epoch: 2970, cost 14.35 s 2023-09-04 14:32:16,454 44k INFO ====> Epoch: 2971, cost 14.17 s 2023-09-04 14:32:22,598 44k INFO Train Epoch: 2972 [38%] 2023-09-04 14:32:22,598 44k INFO Losses: [2.4052813053131104, 2.391026258468628, 6.919356346130371, 13.785411834716797, 0.6082081198692322], step: 62400, lr: 6.897692118830576e-05, reference_loss: 26.109283447265625 2023-09-04 14:32:27,361 44k INFO Saving model and optimizer state at iteration 2972 to ./logs\44k\G_62400.pth 2023-09-04 14:32:27,963 44k INFO Saving model and optimizer state at iteration 2972 to ./logs\44k\D_62400.pth 2023-09-04 14:32:29,207 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_60000.pth 2023-09-04 14:32:29,240 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_60000.pth 2023-09-04 14:32:37,179 44k INFO ====> Epoch: 2972, cost 20.72 s 2023-09-04 14:32:51,217 44k INFO ====> Epoch: 2973, cost 14.04 s 2023-09-04 14:33:05,383 44k INFO ====> Epoch: 2974, cost 14.17 s 2023-09-04 14:33:19,403 44k INFO ====> Epoch: 2975, cost 14.02 s 2023-09-04 14:33:33,399 44k INFO ====> Epoch: 2976, cost 14.00 s 2023-09-04 14:33:47,569 44k INFO ====> Epoch: 2977, cost 14.17 s 2023-09-04 14:34:01,654 44k INFO ====> Epoch: 2978, cost 14.09 s 2023-09-04 14:34:15,851 44k INFO ====> Epoch: 2979, cost 14.20 s 2023-09-04 14:34:29,957 44k INFO ====> Epoch: 2980, cost 14.11 s 2023-09-04 14:34:43,574 44k INFO Train Epoch: 2981 [90%] 2023-09-04 14:34:43,575 44k INFO Losses: [2.1868624687194824, 2.679145097732544, 8.276089668273926, 17.013214111328125, 0.7998656034469604], step: 62600, lr: 6.889936094017264e-05, reference_loss: 30.955177307128906 2023-09-04 14:34:44,256 44k INFO ====> Epoch: 2981, cost 14.30 s 2023-09-04 14:34:58,550 44k INFO ====> Epoch: 2982, cost 14.29 s 2023-09-04 14:35:12,770 44k INFO ====> Epoch: 2983, cost 14.22 s 2023-09-04 14:35:26,871 44k INFO ====> Epoch: 2984, cost 14.10 s 2023-09-04 14:35:40,926 44k INFO ====> Epoch: 2985, cost 14.05 s 2023-09-04 14:35:55,353 44k INFO ====> Epoch: 2986, cost 14.43 s 2023-09-04 14:36:09,656 44k INFO ====> Epoch: 2987, cost 14.30 s 2023-09-04 14:36:23,803 44k INFO ====> Epoch: 2988, cost 14.15 s 2023-09-04 14:36:37,755 44k INFO ====> Epoch: 2989, cost 13.95 s 2023-09-04 14:36:52,006 44k INFO ====> Epoch: 2990, cost 14.25 s 2023-09-04 14:36:59,020 44k INFO Train Epoch: 2991 [43%] 2023-09-04 14:36:59,020 44k INFO Losses: [2.2345802783966064, 2.837185859680176, 7.642685890197754, 16.133642196655273, 0.5788509845733643], step: 62800, lr: 6.881328516771578e-05, reference_loss: 29.42694664001465 2023-09-04 14:37:06,708 44k INFO ====> Epoch: 2991, cost 14.70 s 2023-09-04 14:37:20,790 44k INFO ====> Epoch: 2992, cost 14.08 s 2023-09-04 14:37:34,743 44k INFO ====> Epoch: 2993, cost 13.95 s 2023-09-04 14:37:48,942 44k INFO ====> Epoch: 2994, cost 14.20 s 2023-09-04 14:38:03,091 44k INFO ====> Epoch: 2995, cost 14.15 s 2023-09-04 14:38:17,394 44k INFO ====> Epoch: 2996, cost 14.30 s 2023-09-04 14:38:31,563 44k INFO ====> Epoch: 2997, cost 14.17 s 2023-09-04 14:38:45,857 44k INFO ====> Epoch: 2998, cost 14.29 s 2023-09-04 14:39:00,029 44k INFO ====> Epoch: 2999, cost 14.17 s 2023-09-04 14:39:14,302 44k INFO Train Epoch: 3000 [95%] 2023-09-04 14:39:14,302 44k INFO Losses: [2.6460494995117188, 2.2790653705596924, 5.250690937042236, 15.018375396728516, 0.5021606683731079], step: 63000, lr: 6.873590891808743e-05, reference_loss: 25.69634246826172 2023-09-04 14:39:14,552 44k INFO ====> Epoch: 3000, cost 14.52 s 2023-09-04 14:39:28,641 44k INFO ====> Epoch: 3001, cost 14.09 s 2023-09-04 14:39:42,634 44k INFO ====> Epoch: 3002, cost 13.99 s 2023-09-04 14:39:56,766 44k INFO ====> Epoch: 3003, cost 14.13 s 2023-09-04 14:40:10,954 44k INFO ====> Epoch: 3004, cost 14.19 s 2023-09-04 14:40:25,041 44k INFO ====> Epoch: 3005, cost 14.09 s 2023-09-04 14:40:39,014 44k INFO ====> Epoch: 3006, cost 13.97 s 2023-09-04 14:40:53,191 44k INFO ====> Epoch: 3007, cost 14.18 s 2023-09-04 14:41:07,545 44k INFO ====> Epoch: 3008, cost 14.35 s 2023-09-04 14:41:21,690 44k INFO ====> Epoch: 3009, cost 14.15 s 2023-09-04 14:41:29,196 44k INFO Train Epoch: 3010 [48%] 2023-09-04 14:41:29,196 44k INFO Losses: [2.346545457839966, 2.383263349533081, 8.047333717346191, 17.18179702758789, 0.6572561264038086], step: 63200, lr: 6.86500373457693e-05, reference_loss: 30.616195678710938 2023-09-04 14:41:33,860 44k INFO Saving model and optimizer state at iteration 3010 to ./logs\44k\G_63200.pth 2023-09-04 14:41:34,489 44k INFO Saving model and optimizer state at iteration 3010 to ./logs\44k\D_63200.pth 2023-09-04 14:41:35,559 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_60800.pth 2023-09-04 14:41:35,593 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_60800.pth 2023-09-04 14:41:42,283 44k INFO ====> Epoch: 3010, cost 20.59 s 2023-09-04 14:41:56,588 44k INFO ====> Epoch: 3011, cost 14.31 s 2023-09-04 14:42:11,067 44k INFO ====> Epoch: 3012, cost 14.48 s 2023-09-04 14:42:25,180 44k INFO ====> Epoch: 3013, cost 14.11 s 2023-09-04 14:42:39,307 44k INFO ====> Epoch: 3014, cost 14.13 s 2023-09-04 14:42:53,557 44k INFO ====> Epoch: 3015, cost 14.25 s 2023-09-04 14:43:07,858 44k INFO ====> Epoch: 3016, cost 14.30 s 2023-09-04 14:43:21,999 44k INFO ====> Epoch: 3017, cost 14.14 s 2023-09-04 14:43:35,961 44k INFO ====> Epoch: 3018, cost 13.96 s 2023-09-04 14:43:50,030 44k INFO ====> Epoch: 3019, cost 14.07 s 2023-09-04 14:43:50,746 44k INFO Train Epoch: 3020 [0%] 2023-09-04 14:43:50,746 44k INFO Losses: [2.282219886779785, 2.8572094440460205, 8.327293395996094, 16.694318771362305, 0.6316356062889099], step: 63400, lr: 6.856427305255824e-05, reference_loss: 30.79267692565918 2023-09-04 14:44:04,521 44k INFO ====> Epoch: 3020, cost 14.49 s 2023-09-04 14:44:18,822 44k INFO ====> Epoch: 3021, cost 14.30 s 2023-09-04 14:44:32,984 44k INFO ====> Epoch: 3022, cost 14.16 s 2023-09-04 14:44:47,245 44k INFO ====> Epoch: 3023, cost 14.26 s 2023-09-04 14:45:01,448 44k INFO ====> Epoch: 3024, cost 14.20 s 2023-09-04 14:45:15,825 44k INFO ====> Epoch: 3025, cost 14.38 s 2023-09-04 14:45:29,679 44k INFO ====> Epoch: 3026, cost 13.85 s 2023-09-04 14:45:43,611 44k INFO ====> Epoch: 3027, cost 13.93 s 2023-09-04 14:45:57,768 44k INFO ====> Epoch: 3028, cost 14.16 s 2023-09-04 14:46:06,145 44k INFO Train Epoch: 3029 [52%] 2023-09-04 14:46:06,146 44k INFO Losses: [2.4268603324890137, 2.5204553604125977, 6.720978260040283, 16.470048904418945, 0.7118462324142456], step: 63600, lr: 6.848717680153097e-05, reference_loss: 28.850189208984375 2023-09-04 14:46:12,307 44k INFO ====> Epoch: 3029, cost 14.54 s 2023-09-04 14:46:26,445 44k INFO ====> Epoch: 3030, cost 14.14 s 2023-09-04 14:46:40,548 44k INFO ====> Epoch: 3031, cost 14.10 s 2023-09-04 14:46:54,829 44k INFO ====> Epoch: 3032, cost 14.28 s 2023-09-04 14:47:09,004 44k INFO ====> Epoch: 3033, cost 14.18 s 2023-09-04 14:47:23,142 44k INFO ====> Epoch: 3034, cost 14.14 s 2023-09-04 14:47:37,160 44k INFO ====> Epoch: 3035, cost 14.02 s 2023-09-04 14:47:51,451 44k INFO ====> Epoch: 3036, cost 14.29 s 2023-09-04 14:48:05,836 44k INFO ====> Epoch: 3037, cost 14.38 s 2023-09-04 14:48:20,047 44k INFO ====> Epoch: 3038, cost 14.21 s 2023-09-04 14:48:21,418 44k INFO Train Epoch: 3039 [5%] 2023-09-04 14:48:21,419 44k INFO Losses: [2.236100673675537, 2.6011457443237305, 7.50761079788208, 17.232912063598633, 0.6409692168235779], step: 63800, lr: 6.840161596952704e-05, reference_loss: 30.218738555908203 2023-09-04 14:48:34,642 44k INFO ====> Epoch: 3039, cost 14.60 s 2023-09-04 14:48:48,783 44k INFO ====> Epoch: 3040, cost 14.14 s 2023-09-04 14:49:03,035 44k INFO ====> Epoch: 3041, cost 14.25 s 2023-09-04 14:49:17,050 44k INFO ====> Epoch: 3042, cost 14.01 s 2023-09-04 14:49:31,032 44k INFO ====> Epoch: 3043, cost 13.98 s 2023-09-04 14:49:45,211 44k INFO ====> Epoch: 3044, cost 14.18 s 2023-09-04 14:49:59,382 44k INFO ====> Epoch: 3045, cost 14.17 s 2023-09-04 14:50:13,659 44k INFO ====> Epoch: 3046, cost 14.28 s 2023-09-04 14:50:27,761 44k INFO ====> Epoch: 3047, cost 14.10 s 2023-09-04 14:50:36,652 44k INFO Train Epoch: 3048 [57%] 2023-09-04 14:50:36,653 44k INFO Losses: [2.3298003673553467, 2.496141195297241, 6.394887447357178, 14.995770454406738, 0.8376181125640869], step: 64000, lr: 6.832470261625025e-05, reference_loss: 27.054216384887695 2023-09-04 14:50:41,404 44k INFO Saving model and optimizer state at iteration 3048 to ./logs\44k\G_64000.pth 2023-09-04 14:50:43,396 44k INFO Saving model and optimizer state at iteration 3048 to ./logs\44k\D_64000.pth 2023-09-04 14:50:44,506 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_61600.pth 2023-09-04 14:50:44,547 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_61600.pth 2023-09-04 14:50:49,773 44k INFO ====> Epoch: 3048, cost 22.01 s 2023-09-04 14:51:03,911 44k INFO ====> Epoch: 3049, cost 14.14 s 2023-09-04 14:51:17,864 44k INFO ====> Epoch: 3050, cost 13.95 s 2023-09-04 14:51:31,846 44k INFO ====> Epoch: 3051, cost 13.98 s 2023-09-04 14:51:45,762 44k INFO ====> Epoch: 3052, cost 13.92 s 2023-09-04 14:51:59,976 44k INFO ====> Epoch: 3053, cost 14.21 s 2023-09-04 14:52:13,999 44k INFO ====> Epoch: 3054, cost 14.02 s 2023-09-04 14:52:28,189 44k INFO ====> Epoch: 3055, cost 14.19 s 2023-09-04 14:52:42,343 44k INFO ====> Epoch: 3056, cost 14.15 s 2023-09-04 14:52:56,738 44k INFO ====> Epoch: 3057, cost 14.40 s 2023-09-04 14:52:58,808 44k INFO Train Epoch: 3058 [10%] 2023-09-04 14:52:58,808 44k INFO Losses: [2.598550319671631, 2.040148973464966, 7.183055400848389, 16.018774032592773, 0.6998937726020813], step: 64200, lr: 6.823934476277636e-05, reference_loss: 28.540422439575195 2023-09-04 14:53:11,188 44k INFO ====> Epoch: 3058, cost 14.45 s 2023-09-04 14:53:25,208 44k INFO ====> Epoch: 3059, cost 14.02 s 2023-09-04 14:53:39,330 44k INFO ====> Epoch: 3060, cost 14.12 s 2023-09-04 14:53:53,768 44k INFO ====> Epoch: 3061, cost 14.44 s 2023-09-04 14:54:08,017 44k INFO ====> Epoch: 3062, cost 14.25 s 2023-09-04 14:54:22,263 44k INFO ====> Epoch: 3063, cost 14.25 s 2023-09-04 14:54:36,353 44k INFO ====> Epoch: 3064, cost 14.09 s 2023-09-04 14:54:50,547 44k INFO ====> Epoch: 3065, cost 14.19 s 2023-09-04 14:55:04,783 44k INFO ====> Epoch: 3066, cost 14.24 s 2023-09-04 14:55:14,453 44k INFO Train Epoch: 3067 [62%] 2023-09-04 14:55:14,453 44k INFO Losses: [2.272941827774048, 2.5763087272644043, 8.255782127380371, 16.514705657958984, 0.5990208983421326], step: 64400, lr: 6.816261387335624e-05, reference_loss: 30.218759536743164 2023-09-04 14:55:19,101 44k INFO ====> Epoch: 3067, cost 14.32 s 2023-09-04 14:55:33,080 44k INFO ====> Epoch: 3068, cost 13.98 s 2023-09-04 14:55:47,110 44k INFO ====> Epoch: 3069, cost 14.03 s 2023-09-04 14:56:01,315 44k INFO ====> Epoch: 3070, cost 14.21 s 2023-09-04 14:56:15,421 44k INFO ====> Epoch: 3071, cost 14.11 s 2023-09-04 14:56:29,506 44k INFO ====> Epoch: 3072, cost 14.09 s 2023-09-04 14:56:43,611 44k INFO ====> Epoch: 3073, cost 14.10 s 2023-09-04 14:56:57,758 44k INFO ====> Epoch: 3074, cost 14.15 s 2023-09-04 14:57:12,084 44k INFO ====> Epoch: 3075, cost 14.33 s 2023-09-04 14:57:26,082 44k INFO ====> Epoch: 3076, cost 14.00 s 2023-09-04 14:57:28,813 44k INFO Train Epoch: 3077 [14%] 2023-09-04 14:57:28,813 44k INFO Losses: [2.286182165145874, 2.3538870811462402, 8.076294898986816, 15.450471878051758, 0.6398025751113892], step: 64600, lr: 6.807745851688029e-05, reference_loss: 28.806638717651367 2023-09-04 14:57:40,244 44k INFO ====> Epoch: 3077, cost 14.16 s 2023-09-04 14:57:54,456 44k INFO ====> Epoch: 3078, cost 14.21 s 2023-09-04 14:58:08,653 44k INFO ====> Epoch: 3079, cost 14.20 s 2023-09-04 14:58:22,853 44k INFO ====> Epoch: 3080, cost 14.20 s 2023-09-04 14:58:36,980 44k INFO ====> Epoch: 3081, cost 14.13 s 2023-09-04 14:58:51,143 44k INFO ====> Epoch: 3082, cost 14.16 s 2023-09-04 14:59:05,571 44k INFO ====> Epoch: 3083, cost 14.43 s 2023-09-04 14:59:19,647 44k INFO ====> Epoch: 3084, cost 14.08 s 2023-09-04 14:59:33,685 44k INFO ====> Epoch: 3085, cost 14.04 s 2023-09-04 14:59:43,935 44k INFO Train Epoch: 3086 [67%] 2023-09-04 14:59:43,935 44k INFO Losses: [2.4247989654541016, 2.35713267326355, 7.8556342124938965, 15.874154090881348, 0.6166589260101318], step: 64800, lr: 6.800090965845232e-05, reference_loss: 29.12837791442871 2023-09-04 14:59:48,658 44k INFO Saving model and optimizer state at iteration 3086 to ./logs\44k\G_64800.pth 2023-09-04 14:59:49,271 44k INFO Saving model and optimizer state at iteration 3086 to ./logs\44k\D_64800.pth 2023-09-04 14:59:50,539 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_62400.pth 2023-09-04 14:59:50,572 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_62400.pth 2023-09-04 14:59:54,448 44k INFO ====> Epoch: 3086, cost 20.76 s 2023-09-04 15:00:08,652 44k INFO ====> Epoch: 3087, cost 14.20 s 2023-09-04 15:00:22,644 44k INFO ====> Epoch: 3088, cost 13.99 s 2023-09-04 15:00:36,705 44k INFO ====> Epoch: 3089, cost 14.06 s 2023-09-04 15:00:50,811 44k INFO ====> Epoch: 3090, cost 14.11 s 2023-09-04 15:01:04,946 44k INFO ====> Epoch: 3091, cost 14.13 s 2023-09-04 15:01:19,132 44k INFO ====> Epoch: 3092, cost 14.19 s 2023-09-04 15:01:33,161 44k INFO ====> Epoch: 3093, cost 14.03 s 2023-09-04 15:01:47,169 44k INFO ====> Epoch: 3094, cost 14.01 s 2023-09-04 15:02:01,430 44k INFO ====> Epoch: 3095, cost 14.26 s 2023-09-04 15:02:04,835 44k INFO Train Epoch: 3096 [19%] 2023-09-04 15:02:04,835 44k INFO Losses: [2.562513828277588, 2.244882106781006, 6.830817222595215, 16.494915008544922, 0.6805158257484436], step: 65000, lr: 6.791595631858461e-05, reference_loss: 28.813644409179688 2023-09-04 15:02:15,956 44k INFO ====> Epoch: 3096, cost 14.53 s 2023-09-04 15:02:30,224 44k INFO ====> Epoch: 3097, cost 14.27 s 2023-09-04 15:02:44,299 44k INFO ====> Epoch: 3098, cost 14.07 s 2023-09-04 15:02:58,593 44k INFO ====> Epoch: 3099, cost 14.29 s 2023-09-04 15:03:12,744 44k INFO ====> Epoch: 3100, cost 14.15 s 2023-09-04 15:03:26,827 44k INFO ====> Epoch: 3101, cost 14.08 s 2023-09-04 15:03:40,835 44k INFO ====> Epoch: 3102, cost 14.01 s 2023-09-04 15:03:54,933 44k INFO ====> Epoch: 3103, cost 14.10 s 2023-09-04 15:04:09,240 44k INFO ====> Epoch: 3104, cost 14.31 s 2023-09-04 15:04:20,159 44k INFO Train Epoch: 3105 [71%] 2023-09-04 15:04:20,159 44k INFO Losses: [2.3827319145202637, 2.5970590114593506, 6.425485610961914, 14.651266098022461, 0.6065029501914978], step: 65200, lr: 6.783958905931122e-05, reference_loss: 26.663043975830078 2023-09-04 15:04:23,468 44k INFO ====> Epoch: 3105, cost 14.23 s 2023-09-04 15:04:37,606 44k INFO ====> Epoch: 3106, cost 14.14 s 2023-09-04 15:04:51,798 44k INFO ====> Epoch: 3107, cost 14.19 s 2023-09-04 15:05:05,957 44k INFO ====> Epoch: 3108, cost 14.16 s 2023-09-04 15:05:20,055 44k INFO ====> Epoch: 3109, cost 14.10 s 2023-09-04 15:05:34,120 44k INFO ====> Epoch: 3110, cost 14.07 s 2023-09-04 15:05:48,550 44k INFO ====> Epoch: 3111, cost 14.43 s 2023-09-04 15:06:02,848 44k INFO ====> Epoch: 3112, cost 14.30 s 2023-09-04 15:06:17,142 44k INFO ====> Epoch: 3113, cost 14.29 s 2023-09-04 15:06:31,414 44k INFO ====> Epoch: 3114, cost 14.27 s 2023-09-04 15:06:35,558 44k INFO Train Epoch: 3115 [24%] 2023-09-04 15:06:35,558 44k INFO Losses: [2.369746208190918, 2.4786489009857178, 7.379637718200684, 13.795295715332031, 0.5896556377410889], step: 65400, lr: 6.775483725680169e-05, reference_loss: 26.61298370361328 2023-09-04 15:06:45,947 44k INFO ====> Epoch: 3115, cost 14.53 s 2023-09-04 15:07:00,123 44k INFO ====> Epoch: 3116, cost 14.18 s 2023-09-04 15:07:14,432 44k INFO ====> Epoch: 3117, cost 14.31 s 2023-09-04 15:07:28,344 44k INFO ====> Epoch: 3118, cost 13.91 s 2023-09-04 15:07:42,414 44k INFO ====> Epoch: 3119, cost 14.07 s 2023-09-04 15:07:56,750 44k INFO ====> Epoch: 3120, cost 14.34 s 2023-09-04 15:08:11,016 44k INFO ====> Epoch: 3121, cost 14.27 s 2023-09-04 15:08:25,157 44k INFO ====> Epoch: 3122, cost 14.14 s 2023-09-04 15:08:39,245 44k INFO ====> Epoch: 3123, cost 14.09 s 2023-09-04 15:08:51,014 44k INFO Train Epoch: 3124 [76%] 2023-09-04 15:08:51,014 44k INFO Losses: [2.2313084602355957, 2.538644552230835, 9.110596656799316, 17.140634536743164, 0.6266940236091614], step: 65600, lr: 6.767865116586977e-05, reference_loss: 31.647876739501953 2023-09-04 15:08:55,735 44k INFO Saving model and optimizer state at iteration 3124 to ./logs\44k\G_65600.pth 2023-09-04 15:08:56,410 44k INFO Saving model and optimizer state at iteration 3124 to ./logs\44k\D_65600.pth 2023-09-04 15:08:57,525 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_63200.pth 2023-09-04 15:08:57,558 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_63200.pth 2023-09-04 15:09:00,073 44k INFO ====> Epoch: 3124, cost 20.83 s 2023-09-04 15:09:14,254 44k INFO ====> Epoch: 3125, cost 14.18 s 2023-09-04 15:09:28,246 44k INFO ====> Epoch: 3126, cost 13.99 s 2023-09-04 15:09:42,348 44k INFO ====> Epoch: 3127, cost 14.10 s 2023-09-04 15:09:56,529 44k INFO ====> Epoch: 3128, cost 14.18 s 2023-09-04 15:10:10,881 44k INFO ====> Epoch: 3129, cost 14.35 s 2023-09-04 15:10:24,890 44k INFO ====> Epoch: 3130, cost 14.01 s 2023-09-04 15:10:38,996 44k INFO ====> Epoch: 3131, cost 14.11 s 2023-09-04 15:10:53,270 44k INFO ====> Epoch: 3132, cost 14.27 s 2023-09-04 15:11:07,720 44k INFO ====> Epoch: 3133, cost 14.45 s 2023-09-04 15:11:12,543 44k INFO Train Epoch: 3134 [29%] 2023-09-04 15:11:12,544 44k INFO Losses: [2.2104320526123047, 2.703775405883789, 6.802707672119141, 15.02132511138916, 0.634605348110199], step: 65800, lr: 6.75941004226053e-05, reference_loss: 27.372844696044922 2023-09-04 15:11:21,977 44k INFO ====> Epoch: 3134, cost 14.26 s 2023-09-04 15:11:36,052 44k INFO ====> Epoch: 3135, cost 14.07 s 2023-09-04 15:11:50,397 44k INFO ====> Epoch: 3136, cost 14.35 s 2023-09-04 15:12:04,879 44k INFO ====> Epoch: 3137, cost 14.48 s 2023-09-04 15:12:19,130 44k INFO ====> Epoch: 3138, cost 14.25 s 2023-09-04 15:12:33,356 44k INFO ====> Epoch: 3139, cost 14.23 s 2023-09-04 15:12:47,513 44k INFO ====> Epoch: 3140, cost 14.16 s 2023-09-04 15:13:02,013 44k INFO ====> Epoch: 3141, cost 14.50 s 2023-09-04 15:13:16,203 44k INFO ====> Epoch: 3142, cost 14.19 s 2023-09-04 15:13:28,421 44k INFO Train Epoch: 3143 [81%] 2023-09-04 15:13:28,421 44k INFO Losses: [2.2706050872802734, 2.372178792953491, 8.271270751953125, 15.724489212036133, 0.5823308825492859], step: 66000, lr: 6.751809507022376e-05, reference_loss: 29.220874786376953 2023-09-04 15:13:30,409 44k INFO ====> Epoch: 3143, cost 14.21 s 2023-09-04 15:13:44,454 44k INFO ====> Epoch: 3144, cost 14.04 s 2023-09-04 15:13:58,745 44k INFO ====> Epoch: 3145, cost 14.29 s 2023-09-04 15:14:13,110 44k INFO ====> Epoch: 3146, cost 14.36 s 2023-09-04 15:14:27,246 44k INFO ====> Epoch: 3147, cost 14.14 s 2023-09-04 15:14:41,420 44k INFO ====> Epoch: 3148, cost 14.17 s 2023-09-04 15:14:55,693 44k INFO ====> Epoch: 3149, cost 14.27 s 2023-09-04 15:15:09,807 44k INFO ====> Epoch: 3150, cost 14.11 s 2023-09-04 15:15:23,792 44k INFO ====> Epoch: 3151, cost 13.98 s 2023-09-04 15:15:37,767 44k INFO ====> Epoch: 3152, cost 13.97 s 2023-09-04 15:15:43,233 44k INFO Train Epoch: 3153 [33%] 2023-09-04 15:15:43,233 44k INFO Losses: [2.355316162109375, 2.4278085231781006, 7.2504401206970215, 15.375234603881836, 0.5751050710678101], step: 66200, lr: 6.743374490922548e-05, reference_loss: 27.983905792236328 2023-09-04 15:15:52,077 44k INFO ====> Epoch: 3153, cost 14.31 s 2023-09-04 15:16:06,138 44k INFO ====> Epoch: 3154, cost 14.06 s 2023-09-04 15:16:20,162 44k INFO ====> Epoch: 3155, cost 14.02 s 2023-09-04 15:16:34,139 44k INFO ====> Epoch: 3156, cost 13.98 s 2023-09-04 15:16:48,348 44k INFO ====> Epoch: 3157, cost 14.21 s 2023-09-04 15:17:02,660 44k INFO ====> Epoch: 3158, cost 14.31 s 2023-09-04 15:17:16,892 44k INFO ====> Epoch: 3159, cost 14.23 s 2023-09-04 15:17:30,975 44k INFO ====> Epoch: 3160, cost 14.08 s 2023-09-04 15:17:45,027 44k INFO ====> Epoch: 3161, cost 14.05 s 2023-09-04 15:17:58,384 44k INFO Train Epoch: 3162 [86%] 2023-09-04 15:17:58,384 44k INFO Losses: [2.446087598800659, 2.388197422027588, 8.683156967163086, 16.026927947998047, 0.6371247172355652], step: 66400, lr: 6.735791986662281e-05, reference_loss: 30.181493759155273 2023-09-04 15:18:03,092 44k INFO Saving model and optimizer state at iteration 3162 to ./logs\44k\G_66400.pth 2023-09-04 15:18:03,654 44k INFO Saving model and optimizer state at iteration 3162 to ./logs\44k\D_66400.pth 2023-09-04 15:18:04,850 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_64000.pth 2023-09-04 15:18:04,884 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_64000.pth 2023-09-04 15:18:06,050 44k INFO ====> Epoch: 3162, cost 21.02 s 2023-09-04 15:18:20,155 44k INFO ====> Epoch: 3163, cost 14.11 s 2023-09-04 15:18:34,314 44k INFO ====> Epoch: 3164, cost 14.16 s 2023-09-04 15:18:48,626 44k INFO ====> Epoch: 3165, cost 14.31 s 2023-09-04 15:19:02,878 44k INFO ====> Epoch: 3166, cost 14.25 s 2023-09-04 15:19:17,184 44k INFO ====> Epoch: 3167, cost 14.31 s 2023-09-04 15:19:31,279 44k INFO ====> Epoch: 3168, cost 14.09 s 2023-09-04 15:19:45,420 44k INFO ====> Epoch: 3169, cost 14.14 s 2023-09-04 15:19:59,552 44k INFO ====> Epoch: 3170, cost 14.13 s 2023-09-04 15:20:13,889 44k INFO ====> Epoch: 3171, cost 14.34 s 2023-09-04 15:20:20,038 44k INFO Train Epoch: 3172 [38%] 2023-09-04 15:20:20,038 44k INFO Losses: [2.2933382987976074, 2.6616225242614746, 8.6414155960083, 16.812108993530273, 0.5810635089874268], step: 66600, lr: 6.727376981204337e-05, reference_loss: 30.98954963684082 2023-09-04 15:20:28,277 44k INFO ====> Epoch: 3172, cost 14.39 s 2023-09-04 15:20:42,396 44k INFO ====> Epoch: 3173, cost 14.12 s 2023-09-04 15:20:56,543 44k INFO ====> Epoch: 3174, cost 14.15 s 2023-09-04 15:21:10,613 44k INFO ====> Epoch: 3175, cost 14.07 s 2023-09-04 15:21:24,540 44k INFO ====> Epoch: 3176, cost 13.93 s 2023-09-04 15:21:38,509 44k INFO ====> Epoch: 3177, cost 13.97 s 2023-09-04 15:21:52,721 44k INFO ====> Epoch: 3178, cost 14.21 s 2023-09-04 15:22:06,904 44k INFO ====> Epoch: 3179, cost 14.18 s 2023-09-04 15:22:20,916 44k INFO ====> Epoch: 3180, cost 14.01 s 2023-09-04 15:22:34,446 44k INFO Train Epoch: 3181 [90%] 2023-09-04 15:22:34,447 44k INFO Losses: [2.279913902282715, 2.6280338764190674, 8.118829727172852, 16.18642807006836, 0.5287060141563416], step: 66800, lr: 6.719812465146529e-05, reference_loss: 29.741910934448242 2023-09-04 15:22:35,183 44k INFO ====> Epoch: 3181, cost 14.27 s 2023-09-04 15:22:49,339 44k INFO ====> Epoch: 3182, cost 14.16 s 2023-09-04 15:23:03,574 44k INFO ====> Epoch: 3183, cost 14.24 s 2023-09-04 15:23:17,722 44k INFO ====> Epoch: 3184, cost 14.15 s 2023-09-04 15:23:31,736 44k INFO ====> Epoch: 3185, cost 14.01 s 2023-09-04 15:23:45,879 44k INFO ====> Epoch: 3186, cost 14.14 s 2023-09-04 15:24:00,222 44k INFO ====> Epoch: 3187, cost 14.34 s 2023-09-04 15:24:14,332 44k INFO ====> Epoch: 3188, cost 14.11 s 2023-09-04 15:24:28,580 44k INFO ====> Epoch: 3189, cost 14.25 s 2023-09-04 15:24:42,653 44k INFO ====> Epoch: 3190, cost 14.07 s 2023-09-04 15:24:49,741 44k INFO Train Epoch: 3191 [43%] 2023-09-04 15:24:49,741 44k INFO Losses: [2.0852575302124023, 3.050276279449463, 9.26148509979248, 15.855256080627441, 0.547161340713501], step: 67000, lr: 6.71141742285862e-05, reference_loss: 30.799434661865234 2023-09-04 15:24:57,433 44k INFO ====> Epoch: 3191, cost 14.78 s 2023-09-04 15:25:11,767 44k INFO ====> Epoch: 3192, cost 14.33 s 2023-09-04 15:25:25,756 44k INFO ====> Epoch: 3193, cost 13.99 s 2023-09-04 15:25:39,751 44k INFO ====> Epoch: 3194, cost 14.00 s 2023-09-04 15:25:54,022 44k INFO ====> Epoch: 3195, cost 14.27 s 2023-09-04 15:26:08,285 44k INFO ====> Epoch: 3196, cost 14.26 s 2023-09-04 15:26:22,404 44k INFO ====> Epoch: 3197, cost 14.12 s 2023-09-04 15:26:36,398 44k INFO ====> Epoch: 3198, cost 13.99 s 2023-09-04 15:26:50,571 44k INFO ====> Epoch: 3199, cost 14.17 s 2023-09-04 15:27:04,846 44k INFO Train Epoch: 3200 [95%] 2023-09-04 15:27:04,846 44k INFO Losses: [2.2226154804229736, 2.421651840209961, 6.769601345062256, 16.73322296142578, 0.32388952374458313], step: 67200, lr: 6.703870852329315e-05, reference_loss: 28.470979690551758 2023-09-04 15:27:09,546 44k INFO Saving model and optimizer state at iteration 3200 to ./logs\44k\G_67200.pth 2023-09-04 15:27:10,177 44k INFO Saving model and optimizer state at iteration 3200 to ./logs\44k\D_67200.pth 2023-09-04 15:27:11,402 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_64800.pth 2023-09-04 15:27:11,437 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_64800.pth 2023-09-04 15:27:11,438 44k INFO ====> Epoch: 3200, cost 20.87 s 2023-09-04 15:27:25,456 44k INFO ====> Epoch: 3201, cost 14.02 s 2023-09-04 15:27:39,406 44k INFO ====> Epoch: 3202, cost 13.95 s 2023-09-04 15:27:53,531 44k INFO ====> Epoch: 3203, cost 14.12 s 2023-09-04 15:28:07,865 44k INFO ====> Epoch: 3204, cost 14.33 s 2023-09-04 15:28:22,043 44k INFO ====> Epoch: 3205, cost 14.18 s 2023-09-04 15:28:35,909 44k INFO ====> Epoch: 3206, cost 13.87 s 2023-09-04 15:28:50,083 44k INFO ====> Epoch: 3207, cost 14.17 s 2023-09-04 15:29:04,285 44k INFO ====> Epoch: 3208, cost 14.20 s 2023-09-04 15:29:18,454 44k INFO ====> Epoch: 3209, cost 14.17 s 2023-09-04 15:29:25,930 44k INFO Train Epoch: 3210 [48%] 2023-09-04 15:29:25,930 44k INFO Losses: [2.1760916709899902, 2.8113350868225098, 9.749431610107422, 16.93484878540039, 0.730885922908783], step: 67400, lr: 6.695495725852217e-05, reference_loss: 32.402591705322266 2023-09-04 15:29:32,853 44k INFO ====> Epoch: 3210, cost 14.40 s 2023-09-04 15:29:47,162 44k INFO ====> Epoch: 3211, cost 14.31 s 2023-09-04 15:30:01,726 44k INFO ====> Epoch: 3212, cost 14.56 s 2023-09-04 15:30:16,038 44k INFO ====> Epoch: 3213, cost 14.31 s 2023-09-04 15:30:30,240 44k INFO ====> Epoch: 3214, cost 14.20 s 2023-09-04 15:30:44,410 44k INFO ====> Epoch: 3215, cost 14.17 s 2023-09-04 15:30:58,752 44k INFO ====> Epoch: 3216, cost 14.34 s 2023-09-04 15:31:12,911 44k INFO ====> Epoch: 3217, cost 14.16 s 2023-09-04 15:31:26,871 44k INFO ====> Epoch: 3218, cost 13.96 s 2023-09-04 15:31:40,761 44k INFO ====> Epoch: 3219, cost 13.89 s 2023-09-04 15:31:41,501 44k INFO Train Epoch: 3220 [0%] 2023-09-04 15:31:41,501 44k INFO Losses: [2.5843772888183594, 2.381767749786377, 5.338572978973389, 13.994614601135254, 0.6226898431777954], step: 67600, lr: 6.687131062396418e-05, reference_loss: 24.922021865844727 2023-09-04 15:31:55,334 44k INFO ====> Epoch: 3220, cost 14.57 s 2023-09-04 15:32:09,637 44k INFO ====> Epoch: 3221, cost 14.30 s 2023-09-04 15:32:23,709 44k INFO ====> Epoch: 3222, cost 14.07 s 2023-09-04 15:32:37,828 44k INFO ====> Epoch: 3223, cost 14.12 s 2023-09-04 15:32:51,880 44k INFO ====> Epoch: 3224, cost 14.05 s 2023-09-04 15:33:06,193 44k INFO ====> Epoch: 3225, cost 14.31 s 2023-09-04 15:33:20,156 44k INFO ====> Epoch: 3226, cost 13.96 s 2023-09-04 15:33:34,293 44k INFO ====> Epoch: 3227, cost 14.14 s 2023-09-04 15:33:48,396 44k INFO ====> Epoch: 3228, cost 14.10 s 2023-09-04 15:33:56,782 44k INFO Train Epoch: 3229 [52%] 2023-09-04 15:33:56,783 44k INFO Losses: [2.254384756088257, 2.8517508506774902, 7.029139995574951, 17.107675552368164, 0.5145775079727173], step: 67800, lr: 6.679611800365542e-05, reference_loss: 29.75752830505371 2023-09-04 15:34:02,899 44k INFO ====> Epoch: 3229, cost 14.50 s 2023-09-04 15:34:17,022 44k INFO ====> Epoch: 3230, cost 14.12 s 2023-09-04 15:34:31,255 44k INFO ====> Epoch: 3231, cost 14.23 s 2023-09-04 15:34:45,382 44k INFO ====> Epoch: 3232, cost 14.13 s 2023-09-04 15:34:59,683 44k INFO ====> Epoch: 3233, cost 14.30 s 2023-09-04 15:35:13,806 44k INFO ====> Epoch: 3234, cost 14.12 s 2023-09-04 15:35:27,793 44k INFO ====> Epoch: 3235, cost 13.99 s 2023-09-04 15:35:41,939 44k INFO ====> Epoch: 3236, cost 14.15 s 2023-09-04 15:35:56,325 44k INFO ====> Epoch: 3237, cost 14.39 s 2023-09-04 15:36:10,587 44k INFO ====> Epoch: 3238, cost 14.26 s 2023-09-04 15:36:11,986 44k INFO Train Epoch: 3239 [5%] 2023-09-04 15:36:11,986 44k INFO Losses: [2.680879592895508, 2.2997183799743652, 5.760496616363525, 14.958436965942383, 0.6219096779823303], step: 68000, lr: 6.671266980651937e-05, reference_loss: 26.321441650390625 2023-09-04 15:36:16,631 44k INFO Saving model and optimizer state at iteration 3239 to ./logs\44k\G_68000.pth 2023-09-04 15:36:17,159 44k INFO Saving model and optimizer state at iteration 3239 to ./logs\44k\D_68000.pth 2023-09-04 15:36:18,408 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_65600.pth 2023-09-04 15:36:18,444 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_65600.pth 2023-09-04 15:36:31,230 44k INFO ====> Epoch: 3239, cost 20.64 s 2023-09-04 15:36:45,536 44k INFO ====> Epoch: 3240, cost 14.31 s 2023-09-04 15:36:59,890 44k INFO ====> Epoch: 3241, cost 14.35 s 2023-09-04 15:37:14,135 44k INFO ====> Epoch: 3242, cost 14.25 s 2023-09-04 15:37:28,128 44k INFO ====> Epoch: 3243, cost 13.99 s 2023-09-04 15:37:42,037 44k INFO ====> Epoch: 3244, cost 13.91 s 2023-09-04 15:37:56,364 44k INFO ====> Epoch: 3245, cost 14.33 s 2023-09-04 15:38:10,589 44k INFO ====> Epoch: 3246, cost 14.23 s 2023-09-04 15:38:24,664 44k INFO ====> Epoch: 3247, cost 14.07 s 2023-09-04 15:38:33,571 44k INFO Train Epoch: 3248 [57%] 2023-09-04 15:38:33,571 44k INFO Losses: [2.206430673599243, 2.8829307556152344, 7.366372108459473, 14.233100891113281, 0.6082736253738403], step: 68200, lr: 6.663765556792078e-05, reference_loss: 27.297109603881836 2023-09-04 15:38:39,061 44k INFO ====> Epoch: 3248, cost 14.40 s 2023-09-04 15:38:53,382 44k INFO ====> Epoch: 3249, cost 14.32 s 2023-09-04 15:39:07,581 44k INFO ====> Epoch: 3250, cost 14.20 s 2023-09-04 15:39:21,707 44k INFO ====> Epoch: 3251, cost 14.13 s 2023-09-04 15:39:35,644 44k INFO ====> Epoch: 3252, cost 13.94 s 2023-09-04 15:39:49,703 44k INFO ====> Epoch: 3253, cost 14.06 s 2023-09-04 15:40:04,164 44k INFO ====> Epoch: 3254, cost 14.46 s 2023-09-04 15:40:18,245 44k INFO ====> Epoch: 3255, cost 14.08 s 2023-09-04 15:40:32,240 44k INFO ====> Epoch: 3256, cost 14.00 s 2023-09-04 15:40:46,335 44k INFO ====> Epoch: 3257, cost 14.09 s 2023-09-04 15:40:48,433 44k INFO Train Epoch: 3258 [10%] 2023-09-04 15:40:48,434 44k INFO Losses: [2.457864284515381, 2.3417739868164062, 7.109637260437012, 15.729790687561035, 0.5605888962745667], step: 68400, lr: 6.655440533744762e-05, reference_loss: 28.19965362548828 2023-09-04 15:41:00,791 44k INFO ====> Epoch: 3258, cost 14.46 s 2023-09-04 15:41:14,814 44k INFO ====> Epoch: 3259, cost 14.02 s 2023-09-04 15:41:28,701 44k INFO ====> Epoch: 3260, cost 13.89 s 2023-09-04 15:41:42,652 44k INFO ====> Epoch: 3261, cost 13.95 s 2023-09-04 15:41:57,289 44k INFO ====> Epoch: 3262, cost 14.64 s 2023-09-04 15:42:11,638 44k INFO ====> Epoch: 3263, cost 14.35 s 2023-09-04 15:42:25,679 44k INFO ====> Epoch: 3264, cost 14.04 s 2023-09-04 15:42:39,885 44k INFO ====> Epoch: 3265, cost 14.21 s 2023-09-04 15:42:54,178 44k INFO ====> Epoch: 3266, cost 14.29 s 2023-09-04 15:43:03,865 44k INFO Train Epoch: 3267 [62%] 2023-09-04 15:43:03,866 44k INFO Losses: [2.337523937225342, 2.571751356124878, 7.004936695098877, 14.86264705657959, 0.5082230567932129], step: 68600, lr: 6.647956905737893e-05, reference_loss: 27.28508186340332 2023-09-04 15:43:08,696 44k INFO ====> Epoch: 3267, cost 14.52 s 2023-09-04 15:43:22,660 44k INFO ====> Epoch: 3268, cost 13.96 s 2023-09-04 15:43:36,726 44k INFO ====> Epoch: 3269, cost 14.07 s 2023-09-04 15:43:51,120 44k INFO ====> Epoch: 3270, cost 14.39 s 2023-09-04 15:44:05,318 44k INFO ====> Epoch: 3271, cost 14.20 s 2023-09-04 15:44:19,470 44k INFO ====> Epoch: 3272, cost 14.15 s 2023-09-04 15:44:33,624 44k INFO ====> Epoch: 3273, cost 14.15 s 2023-09-04 15:44:47,616 44k INFO ====> Epoch: 3274, cost 13.99 s 2023-09-04 15:45:01,780 44k INFO ====> Epoch: 3275, cost 14.16 s 2023-09-04 15:45:15,900 44k INFO ====> Epoch: 3276, cost 14.12 s 2023-09-04 15:45:18,611 44k INFO Train Epoch: 3277 [14%] 2023-09-04 15:45:18,611 44k INFO Losses: [2.349041700363159, 2.4127073287963867, 7.241347789764404, 14.714484214782715, 0.6540015935897827], step: 68800, lr: 6.639651632392645e-05, reference_loss: 27.371583938598633 2023-09-04 15:45:23,267 44k INFO Saving model and optimizer state at iteration 3277 to ./logs\44k\G_68800.pth 2023-09-04 15:45:23,894 44k INFO Saving model and optimizer state at iteration 3277 to ./logs\44k\D_68800.pth 2023-09-04 15:45:24,975 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_66400.pth 2023-09-04 15:45:25,007 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_66400.pth 2023-09-04 15:45:36,426 44k INFO ====> Epoch: 3277, cost 20.53 s 2023-09-04 15:45:50,476 44k INFO ====> Epoch: 3278, cost 14.05 s 2023-09-04 15:46:04,879 44k INFO ====> Epoch: 3279, cost 14.40 s 2023-09-04 15:46:19,044 44k INFO ====> Epoch: 3280, cost 14.17 s 2023-09-04 15:46:33,138 44k INFO ====> Epoch: 3281, cost 14.09 s 2023-09-04 15:46:47,243 44k INFO ====> Epoch: 3282, cost 14.10 s 2023-09-04 15:47:01,409 44k INFO ====> Epoch: 3283, cost 14.17 s 2023-09-04 15:47:15,603 44k INFO ====> Epoch: 3284, cost 14.19 s 2023-09-04 15:47:29,572 44k INFO ====> Epoch: 3285, cost 13.97 s 2023-09-04 15:47:39,781 44k INFO Train Epoch: 3286 [67%] 2023-09-04 15:47:39,782 44k INFO Losses: [2.635819435119629, 2.1777703762054443, 6.957490921020508, 15.514810562133789, 0.5831536650657654], step: 69000, lr: 6.63218575802113e-05, reference_loss: 27.86904525756836 2023-09-04 15:47:44,022 44k INFO ====> Epoch: 3286, cost 14.45 s 2023-09-04 15:47:58,510 44k INFO ====> Epoch: 3287, cost 14.49 s 2023-09-04 15:48:12,983 44k INFO ====> Epoch: 3288, cost 14.47 s 2023-09-04 15:48:27,124 44k INFO ====> Epoch: 3289, cost 14.14 s 2023-09-04 15:48:41,281 44k INFO ====> Epoch: 3290, cost 14.16 s 2023-09-04 15:48:55,804 44k INFO ====> Epoch: 3291, cost 14.52 s 2023-09-04 15:49:10,186 44k INFO ====> Epoch: 3292, cost 14.38 s 2023-09-04 15:49:24,160 44k INFO ====> Epoch: 3293, cost 13.97 s 2023-09-04 15:49:38,183 44k INFO ====> Epoch: 3294, cost 14.02 s 2023-09-04 15:49:52,394 44k INFO ====> Epoch: 3295, cost 14.21 s 2023-09-04 15:49:55,853 44k INFO Train Epoch: 3296 [19%] 2023-09-04 15:49:55,854 44k INFO Losses: [2.198759078979492, 2.6705191135406494, 8.408385276794434, 16.198495864868164, 0.49840691685676575], step: 69200, lr: 6.623900187525131e-05, reference_loss: 29.974565505981445 2023-09-04 15:50:06,961 44k INFO ====> Epoch: 3296, cost 14.57 s 2023-09-04 15:50:21,124 44k INFO ====> Epoch: 3297, cost 14.16 s 2023-09-04 15:50:35,299 44k INFO ====> Epoch: 3298, cost 14.18 s 2023-09-04 15:50:49,556 44k INFO ====> Epoch: 3299, cost 14.26 s 2023-09-04 15:51:03,741 44k INFO ====> Epoch: 3300, cost 14.18 s 2023-09-04 15:51:17,667 44k INFO ====> Epoch: 3301, cost 13.93 s 2023-09-04 15:51:31,552 44k INFO ====> Epoch: 3302, cost 13.89 s 2023-09-04 15:51:45,853 44k INFO ====> Epoch: 3303, cost 14.30 s 2023-09-04 15:52:00,139 44k INFO ====> Epoch: 3304, cost 14.29 s 2023-09-04 15:52:11,004 44k INFO Train Epoch: 3305 [71%] 2023-09-04 15:52:11,004 44k INFO Losses: [2.4006617069244385, 2.375087261199951, 7.640836238861084, 14.19642162322998, 0.4353494346141815], step: 69400, lr: 6.616452024671489e-05, reference_loss: 27.048355102539062 2023-09-04 15:52:14,412 44k INFO ====> Epoch: 3305, cost 14.27 s 2023-09-04 15:52:28,477 44k INFO ====> Epoch: 3306, cost 14.06 s 2023-09-04 15:52:42,395 44k INFO ====> Epoch: 3307, cost 13.92 s 2023-09-04 15:52:56,637 44k INFO ====> Epoch: 3308, cost 14.24 s 2023-09-04 15:53:10,765 44k INFO ====> Epoch: 3309, cost 14.13 s 2023-09-04 15:53:24,630 44k INFO ====> Epoch: 3310, cost 13.87 s 2023-09-04 15:53:38,585 44k INFO ====> Epoch: 3311, cost 13.95 s 2023-09-04 15:53:53,008 44k INFO ====> Epoch: 3312, cost 14.42 s 2023-09-04 15:54:07,435 44k INFO ====> Epoch: 3313, cost 14.43 s 2023-09-04 15:54:21,604 44k INFO ====> Epoch: 3314, cost 14.17 s 2023-09-04 15:54:25,662 44k INFO Train Epoch: 3315 [24%] 2023-09-04 15:54:25,662 44k INFO Losses: [2.2363710403442383, 2.747128486633301, 7.973700046539307, 15.485358238220215, 0.6939331889152527], step: 69600, lr: 6.608186110283082e-05, reference_loss: 29.136491775512695 2023-09-04 15:54:30,394 44k INFO Saving model and optimizer state at iteration 3315 to ./logs\44k\G_69600.pth 2023-09-04 15:54:30,938 44k INFO Saving model and optimizer state at iteration 3315 to ./logs\44k\D_69600.pth 2023-09-04 15:54:32,198 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_67200.pth 2023-09-04 15:54:32,230 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_67200.pth 2023-09-04 15:54:42,174 44k INFO ====> Epoch: 3315, cost 20.57 s 2023-09-04 15:54:56,448 44k INFO ====> Epoch: 3316, cost 14.27 s 2023-09-04 15:55:10,729 44k INFO ====> Epoch: 3317, cost 14.28 s 2023-09-04 15:55:24,570 44k INFO ====> Epoch: 3318, cost 13.84 s 2023-09-04 15:55:38,572 44k INFO ====> Epoch: 3319, cost 14.00 s 2023-09-04 15:55:52,859 44k INFO ====> Epoch: 3320, cost 14.29 s 2023-09-04 15:56:07,037 44k INFO ====> Epoch: 3321, cost 14.18 s 2023-09-04 15:56:21,157 44k INFO ====> Epoch: 3322, cost 14.12 s 2023-09-04 15:56:35,410 44k INFO ====> Epoch: 3323, cost 14.25 s 2023-09-04 15:56:47,042 44k INFO Train Epoch: 3324 [76%] 2023-09-04 15:56:47,042 44k INFO Losses: [2.1574320793151855, 2.690308094024658, 8.267850875854492, 17.138629913330078, 0.7505754828453064], step: 69800, lr: 6.600755616929747e-05, reference_loss: 31.004796981811523 2023-09-04 15:56:49,781 44k INFO ====> Epoch: 3324, cost 14.37 s 2023-09-04 15:57:04,017 44k INFO ====> Epoch: 3325, cost 14.24 s 2023-09-04 15:57:18,149 44k INFO ====> Epoch: 3326, cost 14.13 s 2023-09-04 15:57:32,124 44k INFO ====> Epoch: 3327, cost 13.98 s 2023-09-04 15:57:46,101 44k INFO ====> Epoch: 3328, cost 13.98 s 2023-09-04 15:58:00,306 44k INFO ====> Epoch: 3329, cost 14.20 s 2023-09-04 15:58:14,339 44k INFO ====> Epoch: 3330, cost 14.03 s 2023-09-04 15:58:28,367 44k INFO ====> Epoch: 3331, cost 14.03 s 2023-09-04 15:58:42,512 44k INFO ====> Epoch: 3332, cost 14.15 s 2023-09-04 15:58:56,891 44k INFO ====> Epoch: 3333, cost 14.38 s 2023-09-04 15:59:01,642 44k INFO Train Epoch: 3334 [29%] 2023-09-04 15:59:01,643 44k INFO Losses: [2.3981292247772217, 2.3689675331115723, 8.78648853302002, 16.51687240600586, 0.5773444175720215], step: 70000, lr: 6.592509312018164e-05, reference_loss: 30.647802352905273 2023-09-04 15:59:11,275 44k INFO ====> Epoch: 3334, cost 14.38 s 2023-09-04 15:59:25,336 44k INFO ====> Epoch: 3335, cost 14.06 s 2023-09-04 15:59:39,320 44k INFO ====> Epoch: 3336, cost 13.98 s 2023-09-04 15:59:53,709 44k INFO ====> Epoch: 3337, cost 14.39 s 2023-09-04 16:00:08,052 44k INFO ====> Epoch: 3338, cost 14.34 s 2023-09-04 16:00:22,136 44k INFO ====> Epoch: 3339, cost 14.08 s 2023-09-04 16:00:36,191 44k INFO ====> Epoch: 3340, cost 14.06 s 2023-09-04 16:00:50,396 44k INFO ====> Epoch: 3341, cost 14.21 s 2023-09-04 16:01:04,834 44k INFO ====> Epoch: 3342, cost 14.44 s 2023-09-04 16:01:17,379 44k INFO Train Epoch: 3343 [81%] 2023-09-04 16:01:17,379 44k INFO Losses: [2.3735649585723877, 2.705148696899414, 6.849271297454834, 15.108733177185059, 0.7183815836906433], step: 70200, lr: 6.585096446247248e-05, reference_loss: 27.755098342895508 2023-09-04 16:01:19,372 44k INFO ====> Epoch: 3343, cost 14.54 s 2023-09-04 16:01:33,325 44k INFO ====> Epoch: 3344, cost 13.95 s 2023-09-04 16:01:47,460 44k INFO ====> Epoch: 3345, cost 14.14 s 2023-09-04 16:02:01,858 44k INFO ====> Epoch: 3346, cost 14.40 s 2023-09-04 16:02:16,130 44k INFO ====> Epoch: 3347, cost 14.27 s 2023-09-04 16:02:30,207 44k INFO ====> Epoch: 3348, cost 14.08 s 2023-09-04 16:02:44,262 44k INFO ====> Epoch: 3349, cost 14.05 s 2023-09-04 16:02:58,546 44k INFO ====> Epoch: 3350, cost 14.28 s 2023-09-04 16:03:12,704 44k INFO ====> Epoch: 3351, cost 14.16 s 2023-09-04 16:03:26,575 44k INFO ====> Epoch: 3352, cost 13.87 s 2023-09-04 16:03:31,983 44k INFO Train Epoch: 3353 [33%] 2023-09-04 16:03:31,984 44k INFO Losses: [2.394113540649414, 2.3894450664520264, 7.806639671325684, 16.444000244140625, 0.6443874835968018], step: 70400, lr: 6.576869704292332e-05, reference_loss: 29.678586959838867 2023-09-04 16:03:36,652 44k INFO Saving model and optimizer state at iteration 3353 to ./logs\44k\G_70400.pth 2023-09-04 16:03:37,286 44k INFO Saving model and optimizer state at iteration 3353 to ./logs\44k\D_70400.pth 2023-09-04 16:03:39,029 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_68000.pth 2023-09-04 16:03:39,061 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_68000.pth 2023-09-04 16:03:47,748 44k INFO ====> Epoch: 3353, cost 21.17 s 2023-09-04 16:04:01,976 44k INFO ====> Epoch: 3354, cost 14.23 s 2023-09-04 16:04:16,065 44k INFO ====> Epoch: 3355, cost 14.09 s 2023-09-04 16:04:30,072 44k INFO ====> Epoch: 3356, cost 14.01 s 2023-09-04 16:04:44,236 44k INFO ====> Epoch: 3357, cost 14.16 s 2023-09-04 16:04:58,947 44k INFO ====> Epoch: 3358, cost 14.71 s 2023-09-04 16:05:13,052 44k INFO ====> Epoch: 3359, cost 14.10 s 2023-09-04 16:05:27,013 44k INFO ====> Epoch: 3360, cost 13.96 s 2023-09-04 16:05:41,029 44k INFO ====> Epoch: 3361, cost 14.02 s 2023-09-04 16:05:54,423 44k INFO Train Epoch: 3362 [86%] 2023-09-04 16:05:54,423 44k INFO Losses: [2.2477667331695557, 2.6651504039764404, 7.671486854553223, 15.459978103637695, 0.7450478672981262], step: 70600, lr: 6.569474424285396e-05, reference_loss: 28.789430618286133 2023-09-04 16:05:55,799 44k INFO ====> Epoch: 3362, cost 14.77 s 2023-09-04 16:06:10,235 44k INFO ====> Epoch: 3363, cost 14.44 s 2023-09-04 16:06:24,477 44k INFO ====> Epoch: 3364, cost 14.24 s 2023-09-04 16:06:38,563 44k INFO ====> Epoch: 3365, cost 14.09 s 2023-09-04 16:06:52,825 44k INFO ====> Epoch: 3366, cost 14.26 s 2023-09-04 16:07:07,128 44k INFO ====> Epoch: 3367, cost 14.30 s 2023-09-04 16:07:21,355 44k INFO ====> Epoch: 3368, cost 14.23 s 2023-09-04 16:07:35,384 44k INFO ====> Epoch: 3369, cost 14.03 s 2023-09-04 16:07:49,527 44k INFO ====> Epoch: 3370, cost 14.14 s 2023-09-04 16:08:03,832 44k INFO ====> Epoch: 3371, cost 14.30 s 2023-09-04 16:08:10,210 44k INFO Train Epoch: 3372 [38%] 2023-09-04 16:08:10,210 44k INFO Losses: [2.1596243381500244, 2.5414481163024902, 7.95382022857666, 16.072303771972656, 0.5587766170501709], step: 70800, lr: 6.561267198877358e-05, reference_loss: 29.285972595214844 2023-09-04 16:08:18,462 44k INFO ====> Epoch: 3372, cost 14.63 s 2023-09-04 16:08:32,652 44k INFO ====> Epoch: 3373, cost 14.19 s 2023-09-04 16:08:46,778 44k INFO ====> Epoch: 3374, cost 14.13 s 2023-09-04 16:09:01,090 44k INFO ====> Epoch: 3375, cost 14.31 s 2023-09-04 16:09:15,172 44k INFO ====> Epoch: 3376, cost 14.08 s 2023-09-04 16:09:29,162 44k INFO ====> Epoch: 3377, cost 13.99 s 2023-09-04 16:09:43,164 44k INFO ====> Epoch: 3378, cost 14.00 s 2023-09-04 16:09:57,385 44k INFO ====> Epoch: 3379, cost 14.22 s 2023-09-04 16:10:11,496 44k INFO ====> Epoch: 3380, cost 14.11 s 2023-09-04 16:10:25,117 44k INFO Train Epoch: 3381 [90%] 2023-09-04 16:10:25,117 44k INFO Losses: [2.241209030151367, 2.70538330078125, 8.456865310668945, 16.203372955322266, 0.4291916787624359], step: 71000, lr: 6.553889462915162e-05, reference_loss: 30.036022186279297 2023-09-04 16:10:25,783 44k INFO ====> Epoch: 3381, cost 14.29 s 2023-09-04 16:10:39,652 44k INFO ====> Epoch: 3382, cost 13.87 s 2023-09-04 16:10:53,922 44k INFO ====> Epoch: 3383, cost 14.27 s 2023-09-04 16:11:08,220 44k INFO ====> Epoch: 3384, cost 14.30 s 2023-09-04 16:11:22,214 44k INFO ====> Epoch: 3385, cost 13.99 s 2023-09-04 16:11:36,140 44k INFO ====> Epoch: 3386, cost 13.93 s 2023-09-04 16:11:50,417 44k INFO ====> Epoch: 3387, cost 14.28 s 2023-09-04 16:12:04,875 44k INFO ====> Epoch: 3388, cost 14.46 s 2023-09-04 16:12:19,164 44k INFO ====> Epoch: 3389, cost 14.29 s 2023-09-04 16:12:33,216 44k INFO ====> Epoch: 3390, cost 14.05 s 2023-09-04 16:12:40,057 44k INFO Train Epoch: 3391 [43%] 2023-09-04 16:12:40,057 44k INFO Losses: [2.1042981147766113, 2.877593517303467, 7.970700263977051, 15.62519645690918, 0.6858327984809875], step: 71200, lr: 6.545701707754314e-05, reference_loss: 29.263622283935547 2023-09-04 16:12:44,824 44k INFO Saving model and optimizer state at iteration 3391 to ./logs\44k\G_71200.pth 2023-09-04 16:12:45,431 44k INFO Saving model and optimizer state at iteration 3391 to ./logs\44k\D_71200.pth 2023-09-04 16:12:47,158 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_68800.pth 2023-09-04 16:12:47,190 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_68800.pth 2023-09-04 16:12:54,531 44k INFO ====> Epoch: 3391, cost 21.32 s 2023-09-04 16:13:08,911 44k INFO ====> Epoch: 3392, cost 14.38 s 2023-09-04 16:13:22,846 44k INFO ====> Epoch: 3393, cost 13.94 s 2023-09-04 16:13:36,921 44k INFO ====> Epoch: 3394, cost 14.08 s 2023-09-04 16:13:51,078 44k INFO ====> Epoch: 3395, cost 14.16 s 2023-09-04 16:14:05,447 44k INFO ====> Epoch: 3396, cost 14.37 s 2023-09-04 16:14:19,725 44k INFO ====> Epoch: 3397, cost 14.28 s 2023-09-04 16:14:33,737 44k INFO ====> Epoch: 3398, cost 14.01 s 2023-09-04 16:14:47,805 44k INFO ====> Epoch: 3399, cost 14.07 s 2023-09-04 16:15:01,990 44k INFO Train Epoch: 3400 [95%] 2023-09-04 16:15:01,990 44k INFO Losses: [2.5107383728027344, 2.16312837600708, 5.69164514541626, 14.10509204864502, 0.6136434674263], step: 71400, lr: 6.538341474216595e-05, reference_loss: 25.084247589111328 2023-09-04 16:15:02,240 44k INFO ====> Epoch: 3400, cost 14.43 s 2023-09-04 16:15:16,467 44k INFO ====> Epoch: 3401, cost 14.23 s 2023-09-04 16:15:30,399 44k INFO ====> Epoch: 3402, cost 13.93 s 2023-09-04 16:15:44,316 44k INFO ====> Epoch: 3403, cost 13.92 s 2023-09-04 16:15:58,405 44k INFO ====> Epoch: 3404, cost 14.09 s 2023-09-04 16:16:12,534 44k INFO ====> Epoch: 3405, cost 14.13 s 2023-09-04 16:16:26,544 44k INFO ====> Epoch: 3406, cost 14.01 s 2023-09-04 16:16:40,708 44k INFO ====> Epoch: 3407, cost 14.16 s 2023-09-04 16:16:55,052 44k INFO ====> Epoch: 3408, cost 14.34 s 2023-09-04 16:17:09,280 44k INFO ====> Epoch: 3409, cost 14.23 s 2023-09-04 16:17:16,811 44k INFO Train Epoch: 3410 [48%] 2023-09-04 16:17:16,811 44k INFO Losses: [2.261995315551758, 2.5344340801239014, 9.74033260345459, 17.247037887573242, 0.6408017873764038], step: 71600, lr: 6.530173143113084e-05, reference_loss: 32.42460250854492 2023-09-04 16:17:23,652 44k INFO ====> Epoch: 3410, cost 14.37 s 2023-09-04 16:17:37,690 44k INFO ====> Epoch: 3411, cost 14.04 s 2023-09-04 16:17:52,003 44k INFO ====> Epoch: 3412, cost 14.31 s 2023-09-04 16:18:06,381 44k INFO ====> Epoch: 3413, cost 14.38 s 2023-09-04 16:18:20,562 44k INFO ====> Epoch: 3414, cost 14.18 s 2023-09-04 16:18:34,765 44k INFO ====> Epoch: 3415, cost 14.20 s 2023-09-04 16:18:48,964 44k INFO ====> Epoch: 3416, cost 14.20 s 2023-09-04 16:19:03,366 44k INFO ====> Epoch: 3417, cost 14.40 s 2023-09-04 16:19:17,501 44k INFO ====> Epoch: 3418, cost 14.14 s 2023-09-04 16:19:31,505 44k INFO ====> Epoch: 3419, cost 14.00 s 2023-09-04 16:19:32,212 44k INFO Train Epoch: 3420 [0%] 2023-09-04 16:19:32,212 44k INFO Losses: [2.2314538955688477, 2.8300366401672363, 7.470750331878662, 14.769673347473145, 0.6875774264335632], step: 71800, lr: 6.522015016682006e-05, reference_loss: 27.989492416381836 2023-09-04 16:19:45,813 44k INFO ====> Epoch: 3420, cost 14.31 s 2023-09-04 16:20:00,085 44k INFO ====> Epoch: 3421, cost 14.27 s 2023-09-04 16:20:14,304 44k INFO ====> Epoch: 3422, cost 14.22 s 2023-09-04 16:20:28,559 44k INFO ====> Epoch: 3423, cost 14.25 s 2023-09-04 16:20:42,677 44k INFO ====> Epoch: 3424, cost 14.12 s 2023-09-04 16:20:56,869 44k INFO ====> Epoch: 3425, cost 14.19 s 2023-09-04 16:21:11,069 44k INFO ====> Epoch: 3426, cost 14.20 s 2023-09-04 16:21:24,949 44k INFO ====> Epoch: 3427, cost 13.88 s 2023-09-04 16:21:38,970 44k INFO ====> Epoch: 3428, cost 14.02 s 2023-09-04 16:21:47,171 44k INFO Train Epoch: 3429 [52%] 2023-09-04 16:21:47,171 44k INFO Losses: [2.107382297515869, 2.826536178588867, 8.510032653808594, 16.60961151123047, 0.6196405291557312], step: 72000, lr: 6.514681417351867e-05, reference_loss: 30.673202514648438 2023-09-04 16:21:51,952 44k INFO Saving model and optimizer state at iteration 3429 to ./logs\44k\G_72000.pth 2023-09-04 16:21:52,608 44k INFO Saving model and optimizer state at iteration 3429 to ./logs\44k\D_72000.pth 2023-09-04 16:21:53,725 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_69600.pth 2023-09-04 16:21:53,758 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_69600.pth 2023-09-04 16:21:59,786 44k INFO ====> Epoch: 3429, cost 20.82 s 2023-09-04 16:22:13,809 44k INFO ====> Epoch: 3430, cost 14.02 s 2023-09-04 16:22:27,892 44k INFO ====> Epoch: 3431, cost 14.08 s 2023-09-04 16:22:42,071 44k INFO ====> Epoch: 3432, cost 14.18 s 2023-09-04 16:22:56,277 44k INFO ====> Epoch: 3433, cost 14.21 s 2023-09-04 16:23:10,538 44k INFO ====> Epoch: 3434, cost 14.26 s 2023-09-04 16:23:24,413 44k INFO ====> Epoch: 3435, cost 13.88 s 2023-09-04 16:23:38,496 44k INFO ====> Epoch: 3436, cost 14.08 s 2023-09-04 16:23:52,976 44k INFO ====> Epoch: 3437, cost 14.48 s 2023-09-04 16:24:07,247 44k INFO ====> Epoch: 3438, cost 14.27 s 2023-09-04 16:24:08,630 44k INFO Train Epoch: 3439 [5%] 2023-09-04 16:24:08,630 44k INFO Losses: [2.52431321144104, 2.1768314838409424, 7.504888534545898, 14.263060569763184, 0.6410792469978333], step: 72200, lr: 6.506542644689001e-05, reference_loss: 27.110172271728516 2023-09-04 16:24:21,631 44k INFO ====> Epoch: 3439, cost 14.38 s 2023-09-04 16:24:35,629 44k INFO ====> Epoch: 3440, cost 14.00 s 2023-09-04 16:24:50,007 44k INFO ====> Epoch: 3441, cost 14.38 s 2023-09-04 16:25:04,347 44k INFO ====> Epoch: 3442, cost 14.34 s 2023-09-04 16:25:18,478 44k INFO ====> Epoch: 3443, cost 14.13 s 2023-09-04 16:25:32,345 44k INFO ====> Epoch: 3444, cost 13.87 s 2023-09-04 16:25:46,504 44k INFO ====> Epoch: 3445, cost 14.16 s 2023-09-04 16:26:00,723 44k INFO ====> Epoch: 3446, cost 14.22 s 2023-09-04 16:26:15,129 44k INFO ====> Epoch: 3447, cost 14.41 s 2023-09-04 16:26:24,066 44k INFO Train Epoch: 3448 [57%] 2023-09-04 16:26:24,067 44k INFO Losses: [2.3152170181274414, 2.6860668659210205, 8.431669235229492, 17.15541648864746, 0.5295776724815369], step: 72400, lr: 6.499226443076681e-05, reference_loss: 31.11794662475586 2023-09-04 16:26:29,570 44k INFO ====> Epoch: 3448, cost 14.44 s 2023-09-04 16:26:43,664 44k INFO ====> Epoch: 3449, cost 14.09 s 2023-09-04 16:26:57,947 44k INFO ====> Epoch: 3450, cost 14.28 s 2023-09-04 16:27:12,023 44k INFO ====> Epoch: 3451, cost 14.08 s 2023-09-04 16:27:25,936 44k INFO ====> Epoch: 3452, cost 13.91 s 2023-09-04 16:27:39,896 44k INFO ====> Epoch: 3453, cost 13.96 s 2023-09-04 16:27:53,970 44k INFO ====> Epoch: 3454, cost 14.07 s 2023-09-04 16:28:08,078 44k INFO ====> Epoch: 3455, cost 14.11 s 2023-09-04 16:28:22,190 44k INFO ====> Epoch: 3456, cost 14.11 s 2023-09-04 16:28:36,301 44k INFO ====> Epoch: 3457, cost 14.11 s 2023-09-04 16:28:38,377 44k INFO Train Epoch: 3458 [10%] 2023-09-04 16:28:38,378 44k INFO Losses: [2.4543051719665527, 2.3322041034698486, 5.838380813598633, 13.720986366271973, 0.6265600919723511], step: 72600, lr: 6.491106978268502e-05, reference_loss: 24.972436904907227 2023-09-04 16:28:50,815 44k INFO ====> Epoch: 3458, cost 14.51 s 2023-09-04 16:29:05,164 44k INFO ====> Epoch: 3459, cost 14.35 s 2023-09-04 16:29:19,220 44k INFO ====> Epoch: 3460, cost 14.06 s 2023-09-04 16:29:33,162 44k INFO ====> Epoch: 3461, cost 13.94 s 2023-09-04 16:29:47,477 44k INFO ====> Epoch: 3462, cost 14.32 s 2023-09-04 16:30:02,089 44k INFO ====> Epoch: 3463, cost 14.61 s 2023-09-04 16:30:16,371 44k INFO ====> Epoch: 3464, cost 14.28 s 2023-09-04 16:30:30,491 44k INFO ====> Epoch: 3465, cost 14.12 s 2023-09-04 16:30:44,715 44k INFO ====> Epoch: 3466, cost 14.22 s 2023-09-04 16:30:54,541 44k INFO Train Epoch: 3467 [62%] 2023-09-04 16:30:54,541 44k INFO Losses: [2.283374309539795, 2.581929922103882, 6.657918930053711, 14.420478820800781, 0.5696204900741577], step: 72800, lr: 6.483808133100875e-05, reference_loss: 26.513322830200195 2023-09-04 16:30:59,242 44k INFO Saving model and optimizer state at iteration 3467 to ./logs\44k\G_72800.pth 2023-09-04 16:30:59,837 44k INFO Saving model and optimizer state at iteration 3467 to ./logs\44k\D_72800.pth 2023-09-04 16:31:00,973 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_70400.pth 2023-09-04 16:31:01,007 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_70400.pth 2023-09-04 16:31:05,541 44k INFO ====> Epoch: 3467, cost 20.83 s 2023-09-04 16:31:19,623 44k INFO ====> Epoch: 3468, cost 14.08 s 2023-09-04 16:31:33,756 44k INFO ====> Epoch: 3469, cost 14.13 s 2023-09-04 16:31:47,893 44k INFO ====> Epoch: 3470, cost 14.14 s 2023-09-04 16:32:02,201 44k INFO ====> Epoch: 3471, cost 14.31 s 2023-09-04 16:32:16,440 44k INFO ====> Epoch: 3472, cost 14.24 s 2023-09-04 16:32:30,508 44k INFO ====> Epoch: 3473, cost 14.07 s 2023-09-04 16:32:44,788 44k INFO ====> Epoch: 3474, cost 14.28 s 2023-09-04 16:32:59,027 44k INFO ====> Epoch: 3475, cost 14.24 s 2023-09-04 16:33:13,309 44k INFO ====> Epoch: 3476, cost 14.28 s 2023-09-04 16:33:16,003 44k INFO Train Epoch: 3477 [14%] 2023-09-04 16:33:16,004 44k INFO Losses: [2.236480236053467, 2.5649659633636475, 8.747198104858398, 17.1294002532959, 0.7114592790603638], step: 73000, lr: 6.475707930342779e-05, reference_loss: 31.389503479003906 2023-09-04 16:33:27,532 44k INFO ====> Epoch: 3477, cost 14.22 s 2023-09-04 16:33:41,548 44k INFO ====> Epoch: 3478, cost 14.02 s 2023-09-04 16:33:55,767 44k INFO ====> Epoch: 3479, cost 14.22 s 2023-09-04 16:34:09,874 44k INFO ====> Epoch: 3480, cost 14.11 s 2023-09-04 16:34:23,829 44k INFO ====> Epoch: 3481, cost 13.95 s 2023-09-04 16:34:38,053 44k INFO ====> Epoch: 3482, cost 14.22 s 2023-09-04 16:34:52,247 44k INFO ====> Epoch: 3483, cost 14.19 s 2023-09-04 16:35:06,631 44k INFO ====> Epoch: 3484, cost 14.38 s 2023-09-04 16:35:20,651 44k INFO ====> Epoch: 3485, cost 14.02 s 2023-09-04 16:35:30,814 44k INFO Train Epoch: 3486 [67%] 2023-09-04 16:35:30,814 44k INFO Losses: [2.4347500801086426, 2.416917324066162, 8.389039993286133, 16.347084045410156, 0.5715346336364746], step: 73200, lr: 6.46842640044463e-05, reference_loss: 30.159326553344727 2023-09-04 16:35:34,779 44k INFO ====> Epoch: 3486, cost 14.13 s 2023-09-04 16:35:49,156 44k INFO ====> Epoch: 3487, cost 14.38 s 2023-09-04 16:36:03,583 44k INFO ====> Epoch: 3488, cost 14.43 s 2023-09-04 16:36:17,779 44k INFO ====> Epoch: 3489, cost 14.20 s 2023-09-04 16:36:32,016 44k INFO ====> Epoch: 3490, cost 14.24 s 2023-09-04 16:36:46,233 44k INFO ====> Epoch: 3491, cost 14.22 s 2023-09-04 16:37:00,517 44k INFO ====> Epoch: 3492, cost 14.28 s 2023-09-04 16:37:14,791 44k INFO ====> Epoch: 3493, cost 14.27 s 2023-09-04 16:37:28,759 44k INFO ====> Epoch: 3494, cost 13.97 s 2023-09-04 16:37:42,830 44k INFO ====> Epoch: 3495, cost 14.07 s 2023-09-04 16:37:46,414 44k INFO Train Epoch: 3496 [19%] 2023-09-04 16:37:46,414 44k INFO Losses: [2.519745349884033, 2.416539430618286, 7.794506072998047, 15.523859977722168, 0.5158451795578003], step: 73400, lr: 6.460345414040679e-05, reference_loss: 28.77049446105957 2023-09-04 16:37:57,436 44k INFO ====> Epoch: 3496, cost 14.61 s 2023-09-04 16:38:11,730 44k INFO ====> Epoch: 3497, cost 14.29 s 2023-09-04 16:38:25,896 44k INFO ====> Epoch: 3498, cost 14.17 s 2023-09-04 16:38:40,148 44k INFO ====> Epoch: 3499, cost 14.25 s 2023-09-04 16:38:54,196 44k INFO ====> Epoch: 3500, cost 14.05 s 2023-09-04 16:39:08,364 44k INFO ====> Epoch: 3501, cost 14.17 s 2023-09-04 16:39:22,402 44k INFO ====> Epoch: 3502, cost 14.04 s 2023-09-04 16:39:36,479 44k INFO ====> Epoch: 3503, cost 14.08 s 2023-09-04 16:39:50,561 44k INFO ====> Epoch: 3504, cost 14.08 s 2023-09-04 16:40:01,660 44k INFO Train Epoch: 3505 [71%] 2023-09-04 16:40:01,660 44k INFO Losses: [2.4658493995666504, 2.137885332107544, 5.900530815124512, 14.790145874023438, 0.6221328973770142], step: 73600, lr: 6.453081158334474e-05, reference_loss: 25.91654396057129 2023-09-04 16:40:06,365 44k INFO Saving model and optimizer state at iteration 3505 to ./logs\44k\G_73600.pth 2023-09-04 16:40:06,957 44k INFO Saving model and optimizer state at iteration 3505 to ./logs\44k\D_73600.pth 2023-09-04 16:40:08,543 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_71200.pth 2023-09-04 16:40:08,577 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_71200.pth 2023-09-04 16:40:11,755 44k INFO ====> Epoch: 3505, cost 21.19 s 2023-09-04 16:40:25,724 44k INFO ====> Epoch: 3506, cost 13.97 s 2023-09-04 16:40:39,757 44k INFO ====> Epoch: 3507, cost 14.03 s 2023-09-04 16:40:53,998 44k INFO ====> Epoch: 3508, cost 14.24 s 2023-09-04 16:41:08,340 44k INFO ====> Epoch: 3509, cost 14.34 s 2023-09-04 16:41:22,258 44k INFO ====> Epoch: 3510, cost 13.92 s 2023-09-04 16:41:36,231 44k INFO ====> Epoch: 3511, cost 13.97 s 2023-09-04 16:41:50,440 44k INFO ====> Epoch: 3512, cost 14.21 s 2023-09-04 16:42:04,890 44k INFO ====> Epoch: 3513, cost 14.45 s 2023-09-04 16:42:19,131 44k INFO ====> Epoch: 3514, cost 14.24 s 2023-09-04 16:42:23,249 44k INFO Train Epoch: 3515 [24%] 2023-09-04 16:42:23,250 44k INFO Losses: [2.2927703857421875, 2.5347118377685547, 7.104788303375244, 14.407477378845215, 0.6198955178260803], step: 73800, lr: 6.44501934269713e-05, reference_loss: 26.959644317626953 2023-09-04 16:42:33,550 44k INFO ====> Epoch: 3515, cost 14.42 s 2023-09-04 16:42:47,781 44k INFO ====> Epoch: 3516, cost 14.23 s 2023-09-04 16:43:02,196 44k INFO ====> Epoch: 3517, cost 14.41 s 2023-09-04 16:43:16,428 44k INFO ====> Epoch: 3518, cost 14.23 s 2023-09-04 16:43:30,524 44k INFO ====> Epoch: 3519, cost 14.10 s 2023-09-04 16:43:44,558 44k INFO ====> Epoch: 3520, cost 14.03 s 2023-09-04 16:43:58,829 44k INFO ====> Epoch: 3521, cost 14.27 s 2023-09-04 16:44:13,055 44k INFO ====> Epoch: 3522, cost 14.23 s 2023-09-04 16:44:27,188 44k INFO ====> Epoch: 3523, cost 14.13 s 2023-09-04 16:44:38,782 44k INFO Train Epoch: 3524 [76%] 2023-09-04 16:44:38,782 44k INFO Losses: [2.4955713748931885, 2.3854072093963623, 8.849464416503906, 17.392332077026367, 0.5300281047821045], step: 74000, lr: 6.437772320202788e-05, reference_loss: 31.652803421020508 2023-09-04 16:44:41,477 44k INFO ====> Epoch: 3524, cost 14.29 s 2023-09-04 16:44:55,690 44k INFO ====> Epoch: 3525, cost 14.21 s 2023-09-04 16:45:09,992 44k INFO ====> Epoch: 3526, cost 14.30 s 2023-09-04 16:45:23,902 44k INFO ====> Epoch: 3527, cost 13.91 s 2023-09-04 16:45:37,909 44k INFO ====> Epoch: 3528, cost 14.01 s 2023-09-04 16:45:52,082 44k INFO ====> Epoch: 3529, cost 14.17 s 2023-09-04 16:46:06,265 44k INFO ====> Epoch: 3530, cost 14.18 s 2023-09-04 16:46:20,233 44k INFO ====> Epoch: 3531, cost 13.97 s 2023-09-04 16:46:34,214 44k INFO ====> Epoch: 3532, cost 13.98 s 2023-09-04 16:46:48,400 44k INFO ====> Epoch: 3533, cost 14.19 s 2023-09-04 16:46:53,197 44k INFO Train Epoch: 3534 [29%] 2023-09-04 16:46:53,197 44k INFO Losses: [2.10623836517334, 2.649961233139038, 8.786360740661621, 14.63690185546875, 0.4712780714035034], step: 74200, lr: 6.429729629852673e-05, reference_loss: 28.650739669799805 2023-09-04 16:47:02,858 44k INFO ====> Epoch: 3534, cost 14.46 s 2023-09-04 16:47:16,902 44k INFO ====> Epoch: 3535, cost 14.04 s 2023-09-04 16:47:30,796 44k INFO ====> Epoch: 3536, cost 13.89 s 2023-09-04 16:47:44,858 44k INFO ====> Epoch: 3537, cost 14.06 s 2023-09-04 16:47:59,262 44k INFO ====> Epoch: 3538, cost 14.40 s 2023-09-04 16:48:13,602 44k INFO ====> Epoch: 3539, cost 14.34 s 2023-09-04 16:48:27,672 44k INFO ====> Epoch: 3540, cost 14.07 s 2023-09-04 16:48:41,955 44k INFO ====> Epoch: 3541, cost 14.28 s 2023-09-04 16:48:56,332 44k INFO ====> Epoch: 3542, cost 14.38 s 2023-09-04 16:49:08,910 44k INFO Train Epoch: 3543 [81%] 2023-09-04 16:49:08,911 44k INFO Losses: [2.297630548477173, 2.4886152744293213, 7.675053596496582, 14.532998085021973, 0.615508496761322], step: 74400, lr: 6.422499799687324e-05, reference_loss: 27.609806060791016 2023-09-04 16:49:13,661 44k INFO Saving model and optimizer state at iteration 3543 to ./logs\44k\G_74400.pth 2023-09-04 16:49:14,257 44k INFO Saving model and optimizer state at iteration 3543 to ./logs\44k\D_74400.pth 2023-09-04 16:49:15,380 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_72000.pth 2023-09-04 16:49:15,415 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_72000.pth 2023-09-04 16:49:17,221 44k INFO ====> Epoch: 3543, cost 20.89 s 2023-09-04 16:49:31,245 44k INFO ====> Epoch: 3544, cost 14.02 s 2023-09-04 16:49:45,251 44k INFO ====> Epoch: 3545, cost 14.01 s 2023-09-04 16:49:59,563 44k INFO ====> Epoch: 3546, cost 14.31 s 2023-09-04 16:50:13,849 44k INFO ====> Epoch: 3547, cost 14.29 s 2023-09-04 16:50:27,812 44k INFO ====> Epoch: 3548, cost 13.96 s 2023-09-04 16:50:41,993 44k INFO ====> Epoch: 3549, cost 14.18 s 2023-09-04 16:50:56,226 44k INFO ====> Epoch: 3550, cost 14.23 s 2023-09-04 16:51:10,685 44k INFO ====> Epoch: 3551, cost 14.46 s 2023-09-04 16:51:24,700 44k INFO ====> Epoch: 3552, cost 14.02 s 2023-09-04 16:51:30,161 44k INFO Train Epoch: 3553 [33%] 2023-09-04 16:51:30,161 44k INFO Losses: [2.4318814277648926, 2.432279109954834, 7.3765177726745605, 14.862884521484375, 0.6100673675537109], step: 74600, lr: 6.41447618925294e-05, reference_loss: 27.71363067626953 2023-09-04 16:51:39,016 44k INFO ====> Epoch: 3553, cost 14.32 s 2023-09-04 16:51:53,252 44k INFO ====> Epoch: 3554, cost 14.24 s 2023-09-04 16:52:07,388 44k INFO ====> Epoch: 3555, cost 14.14 s 2023-09-04 16:52:21,317 44k INFO ====> Epoch: 3556, cost 13.93 s 2023-09-04 16:52:35,360 44k INFO ====> Epoch: 3557, cost 14.04 s 2023-09-04 16:52:49,494 44k INFO ====> Epoch: 3558, cost 14.13 s 2023-09-04 16:53:03,710 44k INFO ====> Epoch: 3559, cost 14.22 s 2023-09-04 16:53:17,854 44k INFO ====> Epoch: 3560, cost 14.14 s 2023-09-04 16:53:31,829 44k INFO ====> Epoch: 3561, cost 13.97 s 2023-09-04 16:53:44,769 44k INFO Train Epoch: 3562 [86%] 2023-09-04 16:53:44,769 44k INFO Losses: [2.38614821434021, 2.444019079208374, 8.856025695800781, 15.71606731414795, 0.5552471876144409], step: 74800, lr: 6.407263510630707e-05, reference_loss: 29.95750617980957 2023-09-04 16:53:46,104 44k INFO ====> Epoch: 3562, cost 14.28 s 2023-09-04 16:54:00,297 44k INFO ====> Epoch: 3563, cost 14.19 s 2023-09-04 16:54:14,613 44k INFO ====> Epoch: 3564, cost 14.32 s 2023-09-04 16:54:28,683 44k INFO ====> Epoch: 3565, cost 14.07 s 2023-09-04 16:54:43,035 44k INFO ====> Epoch: 3566, cost 14.35 s 2023-09-04 16:54:57,286 44k INFO ====> Epoch: 3567, cost 14.25 s 2023-09-04 16:55:11,653 44k INFO ====> Epoch: 3568, cost 14.37 s 2023-09-04 16:55:25,516 44k INFO ====> Epoch: 3569, cost 13.86 s 2023-09-04 16:55:39,509 44k INFO ====> Epoch: 3570, cost 13.99 s 2023-09-04 16:55:53,900 44k INFO ====> Epoch: 3571, cost 14.39 s 2023-09-04 16:56:00,180 44k INFO Train Epoch: 3572 [38%] 2023-09-04 16:56:00,180 44k INFO Losses: [2.285141944885254, 2.651036500930786, 7.860797882080078, 16.395429611206055, 0.43989071249961853], step: 75000, lr: 6.3992589348482e-05, reference_loss: 29.632295608520508 2023-09-04 16:56:08,424 44k INFO ====> Epoch: 3572, cost 14.52 s 2023-09-04 16:56:22,488 44k INFO ====> Epoch: 3573, cost 14.06 s 2023-09-04 16:56:36,724 44k INFO ====> Epoch: 3574, cost 14.24 s 2023-09-04 16:56:51,006 44k INFO ====> Epoch: 3575, cost 14.28 s 2023-09-04 16:57:05,111 44k INFO ====> Epoch: 3576, cost 14.11 s 2023-09-04 16:57:19,172 44k INFO ====> Epoch: 3577, cost 14.06 s 2023-09-04 16:57:33,112 44k INFO ====> Epoch: 3578, cost 13.94 s 2023-09-04 16:57:47,158 44k INFO ====> Epoch: 3579, cost 14.05 s 2023-09-04 16:58:01,303 44k INFO ====> Epoch: 3580, cost 14.14 s 2023-09-04 16:58:14,931 44k INFO Train Epoch: 3581 [90%] 2023-09-04 16:58:14,932 44k INFO Losses: [2.422166109085083, 2.3762896060943604, 7.4220356941223145, 16.099838256835938, 0.7139405608177185], step: 75200, lr: 6.392063367079963e-05, reference_loss: 29.034269332885742 2023-09-04 16:58:19,757 44k INFO Saving model and optimizer state at iteration 3581 to ./logs\44k\G_75200.pth 2023-09-04 16:58:20,405 44k INFO Saving model and optimizer state at iteration 3581 to ./logs\44k\D_75200.pth 2023-09-04 16:58:21,510 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_72800.pth 2023-09-04 16:58:21,541 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_72800.pth 2023-09-04 16:58:21,998 44k INFO ====> Epoch: 3581, cost 20.70 s 2023-09-04 16:58:35,939 44k INFO ====> Epoch: 3582, cost 13.94 s 2023-09-04 16:58:50,155 44k INFO ====> Epoch: 3583, cost 14.22 s 2023-09-04 16:59:04,454 44k INFO ====> Epoch: 3584, cost 14.30 s 2023-09-04 16:59:18,475 44k INFO ====> Epoch: 3585, cost 14.02 s 2023-09-04 16:59:32,364 44k INFO ====> Epoch: 3586, cost 13.89 s 2023-09-04 16:59:46,497 44k INFO ====> Epoch: 3587, cost 14.13 s 2023-09-04 17:00:00,918 44k INFO ====> Epoch: 3588, cost 14.42 s 2023-09-04 17:00:15,224 44k INFO ====> Epoch: 3589, cost 14.31 s 2023-09-04 17:00:29,456 44k INFO ====> Epoch: 3590, cost 14.23 s 2023-09-04 17:00:36,320 44k INFO Train Epoch: 3591 [43%] 2023-09-04 17:00:36,320 44k INFO Losses: [2.1963515281677246, 2.9045724868774414, 7.982111930847168, 16.35993766784668, 0.5770344138145447], step: 75400, lr: 6.384077780792852e-05, reference_loss: 30.020008087158203 2023-09-04 17:00:43,866 44k INFO ====> Epoch: 3591, cost 14.41 s 2023-09-04 17:00:58,223 44k INFO ====> Epoch: 3592, cost 14.36 s 2023-09-04 17:01:12,543 44k INFO ====> Epoch: 3593, cost 14.32 s 2023-09-04 17:01:26,535 44k INFO ====> Epoch: 3594, cost 13.99 s 2023-09-04 17:01:40,565 44k INFO ====> Epoch: 3595, cost 14.03 s 2023-09-04 17:01:54,868 44k INFO ====> Epoch: 3596, cost 14.30 s 2023-09-04 17:02:09,133 44k INFO ====> Epoch: 3597, cost 14.27 s 2023-09-04 17:02:23,127 44k INFO ====> Epoch: 3598, cost 13.99 s 2023-09-04 17:02:37,355 44k INFO ====> Epoch: 3599, cost 14.23 s 2023-09-04 17:02:51,491 44k INFO Train Epoch: 3600 [95%] 2023-09-04 17:02:51,492 44k INFO Losses: [2.163133382797241, 2.77219557762146, 9.306804656982422, 16.565752029418945, 0.22722725570201874], step: 75600, lr: 6.376899283286019e-05, reference_loss: 31.035114288330078 2023-09-04 17:02:51,744 44k INFO ====> Epoch: 3600, cost 14.39 s 2023-09-04 17:03:06,073 44k INFO ====> Epoch: 3601, cost 14.33 s 2023-09-04 17:03:20,102 44k INFO ====> Epoch: 3602, cost 14.03 s 2023-09-04 17:03:34,220 44k INFO ====> Epoch: 3603, cost 14.12 s 2023-09-04 17:03:48,212 44k INFO ====> Epoch: 3604, cost 13.99 s 2023-09-04 17:04:02,448 44k INFO ====> Epoch: 3605, cost 14.24 s 2023-09-04 17:04:16,560 44k INFO ====> Epoch: 3606, cost 14.11 s 2023-09-04 17:04:30,695 44k INFO ====> Epoch: 3607, cost 14.14 s 2023-09-04 17:04:44,839 44k INFO ====> Epoch: 3608, cost 14.14 s 2023-09-04 17:04:59,177 44k INFO ====> Epoch: 3609, cost 14.34 s 2023-09-04 17:05:06,852 44k INFO Train Epoch: 3610 [48%] 2023-09-04 17:05:06,853 44k INFO Losses: [2.2308552265167236, 2.7327771186828613, 7.622891902923584, 15.014154434204102, 0.5943044424057007], step: 75800, lr: 6.368932641444958e-05, reference_loss: 28.194984436035156 2023-09-04 17:05:13,625 44k INFO ====> Epoch: 3610, cost 14.45 s 2023-09-04 17:05:27,505 44k INFO ====> Epoch: 3611, cost 13.88 s 2023-09-04 17:05:41,520 44k INFO ====> Epoch: 3612, cost 14.02 s 2023-09-04 17:05:55,787 44k INFO ====> Epoch: 3613, cost 14.27 s 2023-09-04 17:06:10,177 44k INFO ====> Epoch: 3614, cost 14.39 s 2023-09-04 17:06:24,270 44k INFO ====> Epoch: 3615, cost 14.09 s 2023-09-04 17:06:38,427 44k INFO ====> Epoch: 3616, cost 14.16 s 2023-09-04 17:06:52,822 44k INFO ====> Epoch: 3617, cost 14.40 s 2023-09-04 17:07:07,304 44k INFO ====> Epoch: 3618, cost 14.48 s 2023-09-04 17:07:21,291 44k INFO ====> Epoch: 3619, cost 13.99 s 2023-09-04 17:07:21,965 44k INFO Train Epoch: 3620 [0%] 2023-09-04 17:07:21,965 44k INFO Losses: [2.2908241748809814, 2.680342674255371, 8.243599891662598, 15.267139434814453, 0.6077830791473389], step: 76000, lr: 6.36097595230652e-05, reference_loss: 29.089689254760742 2023-09-04 17:07:26,660 44k INFO Saving model and optimizer state at iteration 3620 to ./logs\44k\G_76000.pth 2023-09-04 17:07:27,245 44k INFO Saving model and optimizer state at iteration 3620 to ./logs\44k\D_76000.pth 2023-09-04 17:07:28,475 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_73600.pth 2023-09-04 17:07:28,512 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_73600.pth 2023-09-04 17:07:41,835 44k INFO ====> Epoch: 3620, cost 20.54 s 2023-09-04 17:07:56,102 44k INFO ====> Epoch: 3621, cost 14.27 s 2023-09-04 17:08:10,489 44k INFO ====> Epoch: 3622, cost 14.39 s 2023-09-04 17:08:24,787 44k INFO ====> Epoch: 3623, cost 14.30 s 2023-09-04 17:08:38,881 44k INFO ====> Epoch: 3624, cost 14.09 s 2023-09-04 17:08:53,165 44k INFO ====> Epoch: 3625, cost 14.28 s 2023-09-04 17:09:07,654 44k INFO ====> Epoch: 3626, cost 14.49 s 2023-09-04 17:09:21,728 44k INFO ====> Epoch: 3627, cost 14.07 s 2023-09-04 17:09:35,634 44k INFO ====> Epoch: 3628, cost 13.91 s 2023-09-04 17:09:43,886 44k INFO Train Epoch: 3629 [52%] 2023-09-04 17:09:43,886 44k INFO Losses: [2.287794828414917, 2.8432796001434326, 8.509700775146484, 16.161727905273438, 0.6495235562324524], step: 76200, lr: 6.353823431365744e-05, reference_loss: 30.4520263671875 2023-09-04 17:09:49,998 44k INFO ====> Epoch: 3629, cost 14.36 s 2023-09-04 17:10:04,183 44k INFO ====> Epoch: 3630, cost 14.18 s 2023-09-04 17:10:18,376 44k INFO ====> Epoch: 3631, cost 14.19 s 2023-09-04 17:10:32,375 44k INFO ====> Epoch: 3632, cost 14.00 s 2023-09-04 17:10:46,547 44k INFO ====> Epoch: 3633, cost 14.17 s 2023-09-04 17:11:00,761 44k INFO ====> Epoch: 3634, cost 14.21 s 2023-09-04 17:11:14,939 44k INFO ====> Epoch: 3635, cost 14.18 s 2023-09-04 17:11:28,857 44k INFO ====> Epoch: 3636, cost 13.92 s 2023-09-04 17:11:42,927 44k INFO ====> Epoch: 3637, cost 14.07 s 2023-09-04 17:11:57,378 44k INFO ====> Epoch: 3638, cost 14.45 s 2023-09-04 17:11:58,686 44k INFO Train Epoch: 3639 [5%] 2023-09-04 17:11:58,686 44k INFO Losses: [2.312908411026001, 2.566999673843384, 9.597526550292969, 15.724552154541016, 0.534497857093811], step: 76400, lr: 6.345885618119785e-05, reference_loss: 30.73648452758789 2023-09-04 17:12:11,961 44k INFO ====> Epoch: 3639, cost 14.58 s 2023-09-04 17:12:26,140 44k INFO ====> Epoch: 3640, cost 14.18 s 2023-09-04 17:12:40,246 44k INFO ====> Epoch: 3641, cost 14.11 s 2023-09-04 17:12:54,541 44k INFO ====> Epoch: 3642, cost 14.29 s 2023-09-04 17:13:08,913 44k INFO ====> Epoch: 3643, cost 14.37 s 2023-09-04 17:13:22,890 44k INFO ====> Epoch: 3644, cost 13.98 s 2023-09-04 17:13:36,958 44k INFO ====> Epoch: 3645, cost 14.07 s 2023-09-04 17:13:51,119 44k INFO ====> Epoch: 3646, cost 14.16 s 2023-09-04 17:14:05,385 44k INFO ====> Epoch: 3647, cost 14.27 s 2023-09-04 17:14:14,469 44k INFO Train Epoch: 3648 [57%] 2023-09-04 17:14:14,470 44k INFO Losses: [2.130498170852661, 2.9431614875793457, 9.605062484741211, 16.459999084472656, 0.5314245820045471], step: 76600, lr: 6.33875006531913e-05, reference_loss: 31.670146942138672 2023-09-04 17:14:19,896 44k INFO ====> Epoch: 3648, cost 14.51 s 2023-09-04 17:14:34,009 44k INFO ====> Epoch: 3649, cost 14.11 s 2023-09-04 17:14:48,280 44k INFO ====> Epoch: 3650, cost 14.27 s 2023-09-04 17:15:02,547 44k INFO ====> Epoch: 3651, cost 14.27 s 2023-09-04 17:15:16,658 44k INFO ====> Epoch: 3652, cost 14.11 s 2023-09-04 17:15:30,615 44k INFO ====> Epoch: 3653, cost 13.96 s 2023-09-04 17:15:44,578 44k INFO ====> Epoch: 3654, cost 13.96 s 2023-09-04 17:15:58,897 44k INFO ====> Epoch: 3655, cost 14.32 s 2023-09-04 17:16:13,043 44k INFO ====> Epoch: 3656, cost 14.15 s 2023-09-04 17:16:27,103 44k INFO ====> Epoch: 3657, cost 14.06 s 2023-09-04 17:16:29,110 44k INFO Train Epoch: 3658 [10%] 2023-09-04 17:16:29,110 44k INFO Losses: [2.366333246231079, 2.4382123947143555, 8.768205642700195, 14.89173412322998, 0.7398281693458557], step: 76800, lr: 6.3308310831858e-05, reference_loss: 29.204313278198242 2023-09-04 17:16:33,883 44k INFO Saving model and optimizer state at iteration 3658 to ./logs\44k\G_76800.pth 2023-09-04 17:16:34,538 44k INFO Saving model and optimizer state at iteration 3658 to ./logs\44k\D_76800.pth 2023-09-04 17:16:35,643 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_74400.pth 2023-09-04 17:16:35,679 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_74400.pth 2023-09-04 17:16:47,806 44k INFO ====> Epoch: 3658, cost 20.70 s 2023-09-04 17:17:01,920 44k INFO ====> Epoch: 3659, cost 14.11 s 2023-09-04 17:17:16,087 44k INFO ====> Epoch: 3660, cost 14.17 s 2023-09-04 17:17:29,974 44k INFO ====> Epoch: 3661, cost 13.89 s 2023-09-04 17:17:44,026 44k INFO ====> Epoch: 3662, cost 14.05 s 2023-09-04 17:17:58,472 44k INFO ====> Epoch: 3663, cost 14.45 s 2023-09-04 17:18:12,678 44k INFO ====> Epoch: 3664, cost 14.21 s 2023-09-04 17:18:26,703 44k INFO ====> Epoch: 3665, cost 14.03 s 2023-09-04 17:18:40,846 44k INFO ====> Epoch: 3666, cost 14.14 s 2023-09-04 17:18:50,696 44k INFO Train Epoch: 3667 [62%] 2023-09-04 17:18:50,696 44k INFO Losses: [2.278632640838623, 2.6679444313049316, 7.916975021362305, 16.308691024780273, 0.4709794819355011], step: 77000, lr: 6.323712458271241e-05, reference_loss: 29.64322280883789 2023-09-04 17:18:55,408 44k INFO ====> Epoch: 3667, cost 14.56 s 2023-09-04 17:19:09,661 44k INFO ====> Epoch: 3668, cost 14.25 s 2023-09-04 17:19:23,616 44k INFO ====> Epoch: 3669, cost 13.96 s 2023-09-04 17:19:37,506 44k INFO ====> Epoch: 3670, cost 13.89 s 2023-09-04 17:19:51,682 44k INFO ====> Epoch: 3671, cost 14.18 s 2023-09-04 17:20:05,916 44k INFO ====> Epoch: 3672, cost 14.23 s 2023-09-04 17:20:20,016 44k INFO ====> Epoch: 3673, cost 14.10 s 2023-09-04 17:20:34,133 44k INFO ====> Epoch: 3674, cost 14.12 s 2023-09-04 17:20:48,231 44k INFO ====> Epoch: 3675, cost 14.10 s 2023-09-04 17:21:02,468 44k INFO ====> Epoch: 3676, cost 14.24 s 2023-09-04 17:21:05,194 44k INFO Train Epoch: 3677 [14%] 2023-09-04 17:21:05,194 44k INFO Losses: [2.1753416061401367, 2.7221593856811523, 8.995016098022461, 16.830564498901367, 0.5675227046012878], step: 77200, lr: 6.315812262576927e-05, reference_loss: 31.290603637695312 2023-09-04 17:21:16,862 44k INFO ====> Epoch: 3677, cost 14.39 s 2023-09-04 17:21:30,849 44k INFO ====> Epoch: 3678, cost 13.99 s 2023-09-04 17:21:44,881 44k INFO ====> Epoch: 3679, cost 14.03 s 2023-09-04 17:21:58,951 44k INFO ====> Epoch: 3680, cost 14.07 s 2023-09-04 17:22:12,996 44k INFO ====> Epoch: 3681, cost 14.05 s 2023-09-04 17:22:26,990 44k INFO ====> Epoch: 3682, cost 13.99 s 2023-09-04 17:22:41,075 44k INFO ====> Epoch: 3683, cost 14.09 s 2023-09-04 17:22:55,287 44k INFO ====> Epoch: 3684, cost 14.21 s 2023-09-04 17:23:09,554 44k INFO ====> Epoch: 3685, cost 14.27 s 2023-09-04 17:23:19,843 44k INFO Train Epoch: 3686 [67%] 2023-09-04 17:23:19,843 44k INFO Losses: [2.2363033294677734, 2.7252554893493652, 7.3659749031066895, 13.979986190795898, 0.4866308271884918], step: 77400, lr: 6.30871052538993e-05, reference_loss: 26.794151306152344 2023-09-04 17:23:23,877 44k INFO ====> Epoch: 3686, cost 14.32 s 2023-09-04 17:23:37,878 44k INFO ====> Epoch: 3687, cost 14.00 s 2023-09-04 17:23:52,306 44k INFO ====> Epoch: 3688, cost 14.43 s 2023-09-04 17:24:06,739 44k INFO ====> Epoch: 3689, cost 14.43 s 2023-09-04 17:24:20,953 44k INFO ====> Epoch: 3690, cost 14.21 s 2023-09-04 17:24:35,105 44k INFO ====> Epoch: 3691, cost 14.15 s 2023-09-04 17:24:49,282 44k INFO ====> Epoch: 3692, cost 14.18 s 2023-09-04 17:25:03,630 44k INFO ====> Epoch: 3693, cost 14.35 s 2023-09-04 17:25:17,906 44k INFO ====> Epoch: 3694, cost 14.28 s 2023-09-04 17:25:31,858 44k INFO ====> Epoch: 3695, cost 13.95 s 2023-09-04 17:25:35,250 44k INFO Train Epoch: 3696 [19%] 2023-09-04 17:25:35,250 44k INFO Losses: [2.3303844928741455, 2.4571404457092285, 7.688251972198486, 15.68385124206543, 0.7103269100189209], step: 77600, lr: 6.300829071566999e-05, reference_loss: 28.86995506286621 2023-09-04 17:25:39,881 44k INFO Saving model and optimizer state at iteration 3696 to ./logs\44k\G_77600.pth 2023-09-04 17:25:40,584 44k INFO Saving model and optimizer state at iteration 3696 to ./logs\44k\D_77600.pth 2023-09-04 17:25:41,764 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_75200.pth 2023-09-04 17:25:41,799 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_75200.pth 2023-09-04 17:25:52,647 44k INFO ====> Epoch: 3696, cost 20.79 s 2023-09-04 17:26:06,902 44k INFO ====> Epoch: 3697, cost 14.25 s 2023-09-04 17:26:21,130 44k INFO ====> Epoch: 3698, cost 14.23 s 2023-09-04 17:26:35,353 44k INFO ====> Epoch: 3699, cost 14.22 s 2023-09-04 17:26:49,640 44k INFO ====> Epoch: 3700, cost 14.29 s 2023-09-04 17:27:04,027 44k INFO ====> Epoch: 3701, cost 14.39 s 2023-09-04 17:27:18,309 44k INFO ====> Epoch: 3702, cost 14.28 s 2023-09-04 17:27:32,282 44k INFO ====> Epoch: 3703, cost 13.97 s 2023-09-04 17:27:46,259 44k INFO ====> Epoch: 3704, cost 13.98 s 2023-09-04 17:27:57,280 44k INFO Train Epoch: 3705 [71%] 2023-09-04 17:27:57,280 44k INFO Losses: [2.2640576362609863, 2.7058935165405273, 7.647369384765625, 14.557242393493652, 0.7510349154472351], step: 77800, lr: 6.2937441820443e-05, reference_loss: 27.925600051879883 2023-09-04 17:28:00,700 44k INFO ====> Epoch: 3705, cost 14.44 s 2023-09-04 17:28:14,848 44k INFO ====> Epoch: 3706, cost 14.15 s 2023-09-04 17:28:28,777 44k INFO ====> Epoch: 3707, cost 13.93 s 2023-09-04 17:28:42,799 44k INFO ====> Epoch: 3708, cost 14.02 s 2023-09-04 17:28:57,021 44k INFO ====> Epoch: 3709, cost 14.22 s 2023-09-04 17:29:11,374 44k INFO ====> Epoch: 3710, cost 14.35 s 2023-09-04 17:29:25,344 44k INFO ====> Epoch: 3711, cost 13.97 s 2023-09-04 17:29:39,229 44k INFO ====> Epoch: 3712, cost 13.89 s 2023-09-04 17:29:53,533 44k INFO ====> Epoch: 3713, cost 14.30 s 2023-09-04 17:30:07,975 44k INFO ====> Epoch: 3714, cost 14.44 s 2023-09-04 17:30:12,142 44k INFO Train Epoch: 3715 [24%] 2023-09-04 17:30:12,142 44k INFO Losses: [2.1839776039123535, 2.84708309173584, 9.082086563110352, 16.223554611206055, 0.45682424306869507], step: 78000, lr: 6.285881425630847e-05, reference_loss: 30.79352569580078 2023-09-04 17:30:22,508 44k INFO ====> Epoch: 3715, cost 14.53 s 2023-09-04 17:30:36,585 44k INFO ====> Epoch: 3716, cost 14.08 s 2023-09-04 17:30:50,908 44k INFO ====> Epoch: 3717, cost 14.32 s 2023-09-04 17:31:05,247 44k INFO ====> Epoch: 3718, cost 14.34 s 2023-09-04 17:31:19,503 44k INFO ====> Epoch: 3719, cost 14.26 s 2023-09-04 17:31:33,450 44k INFO ====> Epoch: 3720, cost 13.95 s 2023-09-04 17:31:47,495 44k INFO ====> Epoch: 3721, cost 14.04 s 2023-09-04 17:32:01,800 44k INFO ====> Epoch: 3722, cost 14.31 s 2023-09-04 17:32:16,051 44k INFO ====> Epoch: 3723, cost 14.25 s 2023-09-04 17:32:27,734 44k INFO Train Epoch: 3724 [76%] 2023-09-04 17:32:27,734 44k INFO Losses: [1.8446903228759766, 3.2124056816101074, 8.868931770324707, 14.726785659790039, 0.4473535418510437], step: 78200, lr: 6.278813343804228e-05, reference_loss: 29.10016632080078 2023-09-04 17:32:30,406 44k INFO ====> Epoch: 3724, cost 14.36 s 2023-09-04 17:32:44,431 44k INFO ====> Epoch: 3725, cost 14.03 s 2023-09-04 17:32:58,443 44k INFO ====> Epoch: 3726, cost 14.01 s 2023-09-04 17:33:12,808 44k INFO ====> Epoch: 3727, cost 14.37 s 2023-09-04 17:33:26,856 44k INFO ====> Epoch: 3728, cost 14.05 s 2023-09-04 17:33:40,869 44k INFO ====> Epoch: 3729, cost 14.01 s 2023-09-04 17:33:55,053 44k INFO ====> Epoch: 3730, cost 14.18 s 2023-09-04 17:34:09,278 44k INFO ====> Epoch: 3731, cost 14.23 s 2023-09-04 17:34:23,348 44k INFO ====> Epoch: 3732, cost 14.07 s 2023-09-04 17:34:37,528 44k INFO ====> Epoch: 3733, cost 14.18 s 2023-09-04 17:34:42,332 44k INFO Train Epoch: 3734 [29%] 2023-09-04 17:34:42,332 44k INFO Losses: [2.155508041381836, 2.6619608402252197, 11.265131950378418, 17.708351135253906, 0.7483967542648315], step: 78400, lr: 6.270969240443825e-05, reference_loss: 34.53934860229492 2023-09-04 17:34:47,056 44k INFO Saving model and optimizer state at iteration 3734 to ./logs\44k\G_78400.pth 2023-09-04 17:34:47,632 44k INFO Saving model and optimizer state at iteration 3734 to ./logs\44k\D_78400.pth 2023-09-04 17:34:48,904 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_76000.pth 2023-09-04 17:34:48,937 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_76000.pth 2023-09-04 17:34:58,383 44k INFO ====> Epoch: 3734, cost 20.85 s 2023-09-04 17:35:12,490 44k INFO ====> Epoch: 3735, cost 14.11 s 2023-09-04 17:35:26,407 44k INFO ====> Epoch: 3736, cost 13.92 s 2023-09-04 17:35:40,391 44k INFO ====> Epoch: 3737, cost 13.98 s 2023-09-04 17:35:54,755 44k INFO ====> Epoch: 3738, cost 14.36 s 2023-09-04 17:36:09,161 44k INFO ====> Epoch: 3739, cost 14.41 s 2023-09-04 17:36:23,209 44k INFO ====> Epoch: 3740, cost 14.05 s 2023-09-04 17:36:37,440 44k INFO ====> Epoch: 3741, cost 14.23 s 2023-09-04 17:36:51,738 44k INFO ====> Epoch: 3742, cost 14.30 s 2023-09-04 17:37:04,239 44k INFO Train Epoch: 3743 [81%] 2023-09-04 17:37:04,240 44k INFO Losses: [2.3054633140563965, 2.629114866256714, 9.125237464904785, 16.237422943115234, 0.6611042618751526], step: 78600, lr: 6.263917926439885e-05, reference_loss: 30.958341598510742 2023-09-04 17:37:06,358 44k INFO ====> Epoch: 3743, cost 14.62 s 2023-09-04 17:37:20,408 44k INFO ====> Epoch: 3744, cost 14.05 s 2023-09-04 17:37:34,496 44k INFO ====> Epoch: 3745, cost 14.09 s 2023-09-04 17:37:48,716 44k INFO ====> Epoch: 3746, cost 14.22 s 2023-09-04 17:38:02,955 44k INFO ====> Epoch: 3747, cost 14.24 s 2023-09-04 17:38:17,071 44k INFO ====> Epoch: 3748, cost 14.12 s 2023-09-04 17:38:31,116 44k INFO ====> Epoch: 3749, cost 14.05 s 2023-09-04 17:38:45,296 44k INFO ====> Epoch: 3750, cost 14.18 s 2023-09-04 17:38:59,569 44k INFO ====> Epoch: 3751, cost 14.27 s 2023-09-04 17:39:13,771 44k INFO ====> Epoch: 3752, cost 14.20 s 2023-09-04 17:39:19,222 44k INFO Train Epoch: 3753 [33%] 2023-09-04 17:39:19,222 44k INFO Losses: [2.317366123199463, 2.5974273681640625, 8.115147590637207, 17.133331298828125, 0.6618487238883972], step: 78800, lr: 6.256092431881341e-05, reference_loss: 30.82512092590332 2023-09-04 17:39:28,082 44k INFO ====> Epoch: 3753, cost 14.31 s 2023-09-04 17:39:42,147 44k INFO ====> Epoch: 3754, cost 14.07 s 2023-09-04 17:39:56,318 44k INFO ====> Epoch: 3755, cost 14.17 s 2023-09-04 17:40:10,448 44k INFO ====> Epoch: 3756, cost 14.13 s 2023-09-04 17:40:24,385 44k INFO ====> Epoch: 3757, cost 13.94 s 2023-09-04 17:40:38,384 44k INFO ====> Epoch: 3758, cost 14.00 s 2023-09-04 17:40:52,589 44k INFO ====> Epoch: 3759, cost 14.21 s 2023-09-04 17:41:06,824 44k INFO ====> Epoch: 3760, cost 14.23 s 2023-09-04 17:41:20,911 44k INFO ====> Epoch: 3761, cost 14.09 s 2023-09-04 17:41:33,888 44k INFO Train Epoch: 3762 [86%] 2023-09-04 17:41:33,888 44k INFO Losses: [2.3337631225585938, 2.6132924556732178, 9.402887344360352, 15.750829696655273, 0.6846032738685608], step: 79000, lr: 6.249057845921268e-05, reference_loss: 30.785375595092773 2023-09-04 17:41:35,271 44k INFO ====> Epoch: 3762, cost 14.36 s 2023-09-04 17:41:49,646 44k INFO ====> Epoch: 3763, cost 14.37 s 2023-09-04 17:42:04,141 44k INFO ====> Epoch: 3764, cost 14.49 s 2023-09-04 17:42:18,306 44k INFO ====> Epoch: 3765, cost 14.17 s 2023-09-04 17:42:32,256 44k INFO ====> Epoch: 3766, cost 13.95 s 2023-09-04 17:42:46,394 44k INFO ====> Epoch: 3767, cost 14.14 s 2023-09-04 17:43:00,759 44k INFO ====> Epoch: 3768, cost 14.37 s 2023-09-04 17:43:14,980 44k INFO ====> Epoch: 3769, cost 14.22 s 2023-09-04 17:43:28,839 44k INFO ====> Epoch: 3770, cost 13.86 s 2023-09-04 17:43:42,899 44k INFO ====> Epoch: 3771, cost 14.06 s 2023-09-04 17:43:49,181 44k INFO Train Epoch: 3772 [38%] 2023-09-04 17:43:49,182 44k INFO Losses: [2.140780448913574, 3.110398292541504, 8.750774383544922, 14.722481727600098, 0.6342636942863464], step: 79200, lr: 6.24125091601836e-05, reference_loss: 29.35869789123535 2023-09-04 17:43:53,874 44k INFO Saving model and optimizer state at iteration 3772 to ./logs\44k\G_79200.pth 2023-09-04 17:43:54,554 44k INFO Saving model and optimizer state at iteration 3772 to ./logs\44k\D_79200.pth 2023-09-04 17:43:56,124 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_76800.pth 2023-09-04 17:43:56,157 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_76800.pth 2023-09-04 17:44:04,183 44k INFO ====> Epoch: 3772, cost 21.28 s 2023-09-04 17:44:18,245 44k INFO ====> Epoch: 3773, cost 14.06 s 2023-09-04 17:44:32,205 44k INFO ====> Epoch: 3774, cost 13.96 s 2023-09-04 17:44:46,259 44k INFO ====> Epoch: 3775, cost 14.05 s 2023-09-04 17:45:00,535 44k INFO ====> Epoch: 3776, cost 14.28 s 2023-09-04 17:45:14,793 44k INFO ====> Epoch: 3777, cost 14.26 s 2023-09-04 17:45:28,997 44k INFO ====> Epoch: 3778, cost 14.20 s 2023-09-04 17:45:43,085 44k INFO ====> Epoch: 3779, cost 14.09 s 2023-09-04 17:45:57,265 44k INFO ====> Epoch: 3780, cost 14.18 s 2023-09-04 17:46:11,072 44k INFO Train Epoch: 3781 [90%] 2023-09-04 17:46:11,072 44k INFO Losses: [2.4196720123291016, 2.4832048416137695, 8.647767066955566, 16.36680030822754, 0.7844034433364868], step: 79400, lr: 6.234233018417715e-05, reference_loss: 30.701847076416016 2023-09-04 17:46:11,735 44k INFO ====> Epoch: 3781, cost 14.47 s 2023-09-04 17:46:25,796 44k INFO ====> Epoch: 3782, cost 14.06 s 2023-09-04 17:46:39,973 44k INFO ====> Epoch: 3783, cost 14.18 s 2023-09-04 17:46:54,288 44k INFO ====> Epoch: 3784, cost 14.31 s 2023-09-04 17:47:08,576 44k INFO ====> Epoch: 3785, cost 14.29 s 2023-09-04 17:47:22,562 44k INFO ====> Epoch: 3786, cost 13.99 s 2023-09-04 17:47:36,453 44k INFO ====> Epoch: 3787, cost 13.89 s 2023-09-04 17:47:50,979 44k INFO ====> Epoch: 3788, cost 14.53 s 2023-09-04 17:48:05,370 44k INFO ====> Epoch: 3789, cost 14.39 s 2023-09-04 17:48:19,555 44k INFO ====> Epoch: 3790, cost 14.19 s 2023-09-04 17:48:26,363 44k INFO Train Epoch: 3791 [43%] 2023-09-04 17:48:26,363 44k INFO Losses: [2.375896453857422, 2.549269199371338, 7.826227188110352, 15.1459379196167, 0.5598345994949341], step: 79600, lr: 6.226444609128954e-05, reference_loss: 28.45716667175293 2023-09-04 17:48:33,894 44k INFO ====> Epoch: 3791, cost 14.34 s 2023-09-04 17:48:47,927 44k INFO ====> Epoch: 3792, cost 14.03 s 2023-09-04 17:49:02,278 44k INFO ====> Epoch: 3793, cost 14.35 s 2023-09-04 17:49:16,193 44k INFO ====> Epoch: 3794, cost 13.92 s 2023-09-04 17:49:30,314 44k INFO ====> Epoch: 3795, cost 14.12 s 2023-09-04 17:49:44,343 44k INFO ====> Epoch: 3796, cost 14.03 s 2023-09-04 17:49:58,646 44k INFO ====> Epoch: 3797, cost 14.30 s 2023-09-04 17:50:12,711 44k INFO ====> Epoch: 3798, cost 14.06 s 2023-09-04 17:50:26,669 44k INFO ====> Epoch: 3799, cost 13.96 s 2023-09-04 17:50:40,727 44k INFO Train Epoch: 3800 [95%] 2023-09-04 17:50:40,727 44k INFO Losses: [2.4723124504089355, 2.3099899291992188, 7.783596515655518, 14.75489330291748, 0.494904100894928], step: 79800, lr: 6.21944336029744e-05, reference_loss: 27.815696716308594 2023-09-04 17:50:40,977 44k INFO ====> Epoch: 3800, cost 14.31 s 2023-09-04 17:50:55,287 44k INFO ====> Epoch: 3801, cost 14.31 s 2023-09-04 17:51:09,651 44k INFO ====> Epoch: 3802, cost 14.36 s 2023-09-04 17:51:23,557 44k INFO ====> Epoch: 3803, cost 13.91 s 2023-09-04 17:51:37,547 44k INFO ====> Epoch: 3804, cost 13.99 s 2023-09-04 17:51:51,614 44k INFO ====> Epoch: 3805, cost 14.07 s 2023-09-04 17:52:05,687 44k INFO ====> Epoch: 3806, cost 14.07 s 2023-09-04 17:52:19,673 44k INFO ====> Epoch: 3807, cost 13.99 s 2023-09-04 17:52:33,692 44k INFO ====> Epoch: 3808, cost 14.02 s 2023-09-04 17:52:47,734 44k INFO ====> Epoch: 3809, cost 14.04 s 2023-09-04 17:52:55,276 44k INFO Train Epoch: 3810 [48%] 2023-09-04 17:52:55,276 44k INFO Losses: [2.1457765102386475, 2.9071462154388428, 10.662571907043457, 17.739429473876953, 0.6693932414054871], step: 80000, lr: 6.211673427685816e-05, reference_loss: 34.12431716918945 2023-09-04 17:53:00,076 44k INFO Saving model and optimizer state at iteration 3810 to ./logs\44k\G_80000.pth 2023-09-04 17:53:00,667 44k INFO Saving model and optimizer state at iteration 3810 to ./logs\44k\D_80000.pth 2023-09-04 17:53:01,885 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_77600.pth 2023-09-04 17:53:01,920 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_77600.pth 2023-09-04 17:53:08,618 44k INFO ====> Epoch: 3810, cost 20.88 s 2023-09-04 17:53:22,514 44k INFO ====> Epoch: 3811, cost 13.90 s 2023-09-04 17:53:36,452 44k INFO ====> Epoch: 3812, cost 13.94 s 2023-09-04 17:53:50,898 44k INFO ====> Epoch: 3813, cost 14.45 s 2023-09-04 17:54:05,217 44k INFO ====> Epoch: 3814, cost 14.32 s 2023-09-04 17:54:19,480 44k INFO ====> Epoch: 3815, cost 14.26 s 2023-09-04 17:54:33,582 44k INFO ====> Epoch: 3816, cost 14.10 s 2023-09-04 17:54:47,868 44k INFO ====> Epoch: 3817, cost 14.29 s 2023-09-04 17:55:02,277 44k INFO ====> Epoch: 3818, cost 14.41 s 2023-09-04 17:55:16,518 44k INFO ====> Epoch: 3819, cost 14.24 s 2023-09-04 17:55:17,210 44k INFO Train Epoch: 3820 [0%] 2023-09-04 17:55:17,210 44k INFO Losses: [2.1694629192352295, 2.846660852432251, 7.493775367736816, 15.493621826171875, 0.5797827839851379], step: 80200, lr: 6.203913202028542e-05, reference_loss: 28.583303451538086 2023-09-04 17:55:30,649 44k INFO ====> Epoch: 3820, cost 14.13 s 2023-09-04 17:55:44,727 44k INFO ====> Epoch: 3821, cost 14.08 s 2023-09-04 17:55:58,904 44k INFO ====> Epoch: 3822, cost 14.18 s 2023-09-04 17:56:13,185 44k INFO ====> Epoch: 3823, cost 14.28 s 2023-09-04 17:56:27,387 44k INFO ====> Epoch: 3824, cost 14.20 s 2023-09-04 17:56:41,565 44k INFO ====> Epoch: 3825, cost 14.18 s 2023-09-04 17:56:55,877 44k INFO ====> Epoch: 3826, cost 14.31 s 2023-09-04 17:57:10,217 44k INFO ====> Epoch: 3827, cost 14.34 s 2023-09-04 17:57:24,197 44k INFO ====> Epoch: 3828, cost 13.98 s 2023-09-04 17:57:32,313 44k INFO Train Epoch: 3829 [52%] 2023-09-04 17:57:32,314 44k INFO Losses: [2.375890016555786, 2.4841458797454834, 7.336053371429443, 16.72663688659668, 0.6999222040176392], step: 80400, lr: 6.196937288359794e-05, reference_loss: 29.622648239135742 2023-09-04 17:57:38,456 44k INFO ====> Epoch: 3829, cost 14.26 s 2023-09-04 17:57:52,756 44k INFO ====> Epoch: 3830, cost 14.30 s 2023-09-04 17:58:06,930 44k INFO ====> Epoch: 3831, cost 14.17 s 2023-09-04 17:58:20,978 44k INFO ====> Epoch: 3832, cost 14.05 s 2023-09-04 17:58:34,966 44k INFO ====> Epoch: 3833, cost 13.99 s 2023-09-04 17:58:49,226 44k INFO ====> Epoch: 3834, cost 14.26 s 2023-09-04 17:59:03,593 44k INFO ====> Epoch: 3835, cost 14.37 s 2023-09-04 17:59:17,769 44k INFO ====> Epoch: 3836, cost 14.18 s 2023-09-04 17:59:31,864 44k INFO ====> Epoch: 3837, cost 14.10 s 2023-09-04 17:59:46,098 44k INFO ====> Epoch: 3838, cost 14.23 s 2023-09-04 17:59:47,525 44k INFO Train Epoch: 3839 [5%] 2023-09-04 17:59:47,525 44k INFO Losses: [2.21334171295166, 2.933032751083374, 8.306197166442871, 17.00074005126953, 0.535088300704956], step: 80600, lr: 6.189195472518783e-05, reference_loss: 30.988399505615234 2023-09-04 18:00:00,691 44k INFO ====> Epoch: 3839, cost 14.59 s 2023-09-04 18:00:14,995 44k INFO ====> Epoch: 3840, cost 14.30 s 2023-09-04 18:00:29,073 44k INFO ====> Epoch: 3841, cost 14.08 s 2023-09-04 18:00:43,152 44k INFO ====> Epoch: 3842, cost 14.08 s 2023-09-04 18:00:57,363 44k INFO ====> Epoch: 3843, cost 14.21 s 2023-09-04 18:01:11,768 44k INFO ====> Epoch: 3844, cost 14.40 s 2023-09-04 18:01:25,716 44k INFO ====> Epoch: 3845, cost 13.95 s 2023-09-04 18:01:39,684 44k INFO ====> Epoch: 3846, cost 13.97 s 2023-09-04 18:01:54,040 44k INFO ====> Epoch: 3847, cost 14.36 s 2023-09-04 18:02:03,160 44k INFO Train Epoch: 3848 [57%] 2023-09-04 18:02:03,160 44k INFO Losses: [2.0754477977752686, 2.8212409019470215, 8.641024589538574, 15.480819702148438, 0.7884213328361511], step: 80800, lr: 6.182236108019425e-05, reference_loss: 29.80695343017578 2023-09-04 18:02:07,922 44k INFO Saving model and optimizer state at iteration 3848 to ./logs\44k\G_80800.pth 2023-09-04 18:02:08,633 44k INFO Saving model and optimizer state at iteration 3848 to ./logs\44k\D_80800.pth 2023-09-04 18:02:10,344 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_78400.pth 2023-09-04 18:02:10,376 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_78400.pth 2023-09-04 18:02:15,582 44k INFO ====> Epoch: 3848, cost 21.54 s 2023-09-04 18:02:29,649 44k INFO ====> Epoch: 3849, cost 14.07 s 2023-09-04 18:02:43,717 44k INFO ====> Epoch: 3850, cost 14.07 s 2023-09-04 18:02:57,847 44k INFO ====> Epoch: 3851, cost 14.13 s 2023-09-04 18:03:12,138 44k INFO ====> Epoch: 3852, cost 14.29 s 2023-09-04 18:03:26,137 44k INFO ====> Epoch: 3853, cost 14.00 s 2023-09-04 18:03:40,154 44k INFO ====> Epoch: 3854, cost 14.02 s 2023-09-04 18:03:54,436 44k INFO ====> Epoch: 3855, cost 14.28 s 2023-09-04 18:04:08,465 44k INFO ====> Epoch: 3856, cost 14.03 s 2023-09-04 18:04:22,420 44k INFO ====> Epoch: 3857, cost 13.95 s 2023-09-04 18:04:24,475 44k INFO Train Epoch: 3858 [10%] 2023-09-04 18:04:24,475 44k INFO Losses: [2.4183547496795654, 2.3597090244293213, 7.765594005584717, 15.484542846679688, 0.6373187303543091], step: 81000, lr: 6.174512658320517e-05, reference_loss: 28.66551971435547 2023-09-04 18:04:36,767 44k INFO ====> Epoch: 3858, cost 14.35 s 2023-09-04 18:04:50,982 44k INFO ====> Epoch: 3859, cost 14.21 s 2023-09-04 18:05:05,353 44k INFO ====> Epoch: 3860, cost 14.37 s 2023-09-04 18:05:19,367 44k INFO ====> Epoch: 3861, cost 14.01 s 2023-09-04 18:05:33,272 44k INFO ====> Epoch: 3862, cost 13.90 s 2023-09-04 18:05:47,612 44k INFO ====> Epoch: 3863, cost 14.34 s 2023-09-04 18:06:01,960 44k INFO ====> Epoch: 3864, cost 14.35 s 2023-09-04 18:06:16,334 44k INFO ====> Epoch: 3865, cost 14.37 s 2023-09-04 18:06:30,487 44k INFO ====> Epoch: 3866, cost 14.15 s 2023-09-04 18:06:40,125 44k INFO Train Epoch: 3867 [62%] 2023-09-04 18:06:40,125 44k INFO Losses: [2.220396041870117, 2.707759141921997, 8.804780006408691, 17.492727279663086, 0.5518120527267456], step: 81200, lr: 6.167569803730456e-05, reference_loss: 31.77747344970703 2023-09-04 18:06:45,030 44k INFO ====> Epoch: 3867, cost 14.54 s 2023-09-04 18:06:59,370 44k INFO ====> Epoch: 3868, cost 14.34 s 2023-09-04 18:07:13,548 44k INFO ====> Epoch: 3869, cost 14.18 s 2023-09-04 18:07:27,537 44k INFO ====> Epoch: 3870, cost 13.99 s 2023-09-04 18:07:41,486 44k INFO ====> Epoch: 3871, cost 13.95 s 2023-09-04 18:07:55,697 44k INFO ====> Epoch: 3872, cost 14.21 s 2023-09-04 18:08:10,010 44k INFO ====> Epoch: 3873, cost 14.31 s 2023-09-04 18:08:24,016 44k INFO ====> Epoch: 3874, cost 14.01 s 2023-09-04 18:08:38,121 44k INFO ====> Epoch: 3875, cost 14.10 s 2023-09-04 18:08:52,227 44k INFO ====> Epoch: 3876, cost 14.11 s 2023-09-04 18:08:55,039 44k INFO Train Epoch: 3877 [14%] 2023-09-04 18:08:55,040 44k INFO Losses: [2.0959267616271973, 2.852778911590576, 9.6124906539917, 15.406866073608398, 0.647681713104248], step: 81400, lr: 6.159864676603102e-05, reference_loss: 30.615745544433594 2023-09-04 18:09:06,937 44k INFO ====> Epoch: 3877, cost 14.71 s 2023-09-04 18:09:21,018 44k INFO ====> Epoch: 3878, cost 14.08 s 2023-09-04 18:09:34,888 44k INFO ====> Epoch: 3879, cost 13.87 s 2023-09-04 18:09:48,993 44k INFO ====> Epoch: 3880, cost 14.11 s 2023-09-04 18:10:03,203 44k INFO ====> Epoch: 3881, cost 14.21 s 2023-09-04 18:10:17,305 44k INFO ====> Epoch: 3882, cost 14.10 s 2023-09-04 18:10:31,365 44k INFO ====> Epoch: 3883, cost 14.06 s 2023-09-04 18:10:45,387 44k INFO ====> Epoch: 3884, cost 14.02 s 2023-09-04 18:10:59,695 44k INFO ====> Epoch: 3885, cost 14.31 s 2023-09-04 18:11:09,999 44k INFO Train Epoch: 3886 [67%] 2023-09-04 18:11:09,999 44k INFO Losses: [2.3030526638031006, 2.4869883060455322, 6.836632251739502, 16.616777420043945, 0.6171872019767761], step: 81600, lr: 6.152938292755391e-05, reference_loss: 28.860637664794922 2023-09-04 18:11:14,734 44k INFO Saving model and optimizer state at iteration 3886 to ./logs\44k\G_81600.pth 2023-09-04 18:11:15,369 44k INFO Saving model and optimizer state at iteration 3886 to ./logs\44k\D_81600.pth 2023-09-04 18:11:16,436 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_79200.pth 2023-09-04 18:11:16,472 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_79200.pth 2023-09-04 18:11:20,295 44k INFO ====> Epoch: 3886, cost 20.60 s 2023-09-04 18:11:34,382 44k INFO ====> Epoch: 3887, cost 14.09 s 2023-09-04 18:11:48,660 44k INFO ====> Epoch: 3888, cost 14.28 s 2023-09-04 18:12:02,937 44k INFO ====> Epoch: 3889, cost 14.28 s 2023-09-04 18:12:17,168 44k INFO ====> Epoch: 3890, cost 14.23 s 2023-09-04 18:12:31,367 44k INFO ====> Epoch: 3891, cost 14.20 s 2023-09-04 18:12:45,483 44k INFO ====> Epoch: 3892, cost 14.12 s 2023-09-04 18:12:59,840 44k INFO ====> Epoch: 3893, cost 14.36 s 2023-09-04 18:13:13,967 44k INFO ====> Epoch: 3894, cost 14.13 s 2023-09-04 18:13:28,467 44k INFO ====> Epoch: 3895, cost 14.50 s 2023-09-04 18:13:31,989 44k INFO Train Epoch: 3896 [19%] 2023-09-04 18:13:31,989 44k INFO Losses: [2.3313279151916504, 2.7603068351745605, 7.889882564544678, 16.585765838623047, 0.6995483040809631], step: 81800, lr: 6.145251444732401e-05, reference_loss: 30.26683235168457 2023-09-04 18:13:43,449 44k INFO ====> Epoch: 3896, cost 14.98 s 2023-09-04 18:13:57,629 44k INFO ====> Epoch: 3897, cost 14.18 s 2023-09-04 18:14:11,743 44k INFO ====> Epoch: 3898, cost 14.11 s 2023-09-04 18:14:25,803 44k INFO ====> Epoch: 3899, cost 14.06 s 2023-09-04 18:14:39,997 44k INFO ====> Epoch: 3900, cost 14.19 s 2023-09-04 18:14:54,121 44k INFO ====> Epoch: 3901, cost 14.12 s 2023-09-04 18:15:08,187 44k INFO ====> Epoch: 3902, cost 14.07 s 2023-09-04 18:15:22,400 44k INFO ====> Epoch: 3903, cost 14.21 s 2023-09-04 18:15:36,464 44k INFO ====> Epoch: 3904, cost 14.06 s 2023-09-04 18:15:47,493 44k INFO Train Epoch: 3905 [71%] 2023-09-04 18:15:47,493 44k INFO Losses: [2.1727521419525146, 2.637749671936035, 9.526050567626953, 14.816325187683105, 0.601193904876709], step: 82000, lr: 6.138341492552999e-05, reference_loss: 29.754072189331055 2023-09-04 18:15:50,935 44k INFO ====> Epoch: 3905, cost 14.47 s 2023-09-04 18:16:05,105 44k INFO ====> Epoch: 3906, cost 14.17 s 2023-09-04 18:16:19,460 44k INFO ====> Epoch: 3907, cost 14.35 s 2023-09-04 18:16:33,605 44k INFO ====> Epoch: 3908, cost 14.15 s 2023-09-04 18:16:47,927 44k INFO ====> Epoch: 3909, cost 14.32 s 2023-09-04 18:17:02,223 44k INFO ====> Epoch: 3910, cost 14.30 s 2023-09-04 18:17:16,583 44k INFO ====> Epoch: 3911, cost 14.36 s 2023-09-04 18:17:30,596 44k INFO ====> Epoch: 3912, cost 14.01 s 2023-09-04 18:17:44,680 44k INFO ====> Epoch: 3913, cost 14.08 s 2023-09-04 18:17:59,086 44k INFO ====> Epoch: 3914, cost 14.41 s 2023-09-04 18:18:03,294 44k INFO Train Epoch: 3915 [24%] 2023-09-04 18:18:03,295 44k INFO Losses: [2.330850601196289, 2.4567203521728516, 8.838297843933105, 14.53520679473877, 0.5744990110397339], step: 82200, lr: 6.130672880270308e-05, reference_loss: 28.73557472229004 2023-09-04 18:18:13,699 44k INFO ====> Epoch: 3915, cost 14.61 s 2023-09-04 18:18:27,896 44k INFO ====> Epoch: 3916, cost 14.20 s 2023-09-04 18:18:42,051 44k INFO ====> Epoch: 3917, cost 14.15 s 2023-09-04 18:18:56,485 44k INFO ====> Epoch: 3918, cost 14.43 s 2023-09-04 18:19:10,933 44k INFO ====> Epoch: 3919, cost 14.45 s 2023-09-04 18:19:25,010 44k INFO ====> Epoch: 3920, cost 14.08 s 2023-09-04 18:19:39,094 44k INFO ====> Epoch: 3921, cost 14.08 s 2023-09-04 18:19:53,322 44k INFO ====> Epoch: 3922, cost 14.23 s 2023-09-04 18:20:07,549 44k INFO ====> Epoch: 3923, cost 14.23 s 2023-09-04 18:20:19,226 44k INFO Train Epoch: 3924 [76%] 2023-09-04 18:20:19,226 44k INFO Losses: [2.2940242290496826, 2.378312349319458, 9.373795509338379, 17.699207305908203, 0.6186993718147278], step: 82400, lr: 6.123779320777873e-05, reference_loss: 32.364036560058594 2023-09-04 18:20:24,037 44k INFO Saving model and optimizer state at iteration 3924 to ./logs\44k\G_82400.pth 2023-09-04 18:20:24,607 44k INFO Saving model and optimizer state at iteration 3924 to ./logs\44k\D_82400.pth 2023-09-04 18:20:25,737 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_80000.pth 2023-09-04 18:20:25,770 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_80000.pth 2023-09-04 18:20:28,308 44k INFO ====> Epoch: 3924, cost 20.76 s 2023-09-04 18:20:42,506 44k INFO ====> Epoch: 3925, cost 14.20 s 2023-09-04 18:20:56,898 44k INFO ====> Epoch: 3926, cost 14.39 s 2023-09-04 18:21:11,122 44k INFO ====> Epoch: 3927, cost 14.22 s 2023-09-04 18:21:25,004 44k INFO ====> Epoch: 3928, cost 13.88 s 2023-09-04 18:21:38,924 44k INFO ====> Epoch: 3929, cost 13.92 s 2023-09-04 18:21:53,015 44k INFO ====> Epoch: 3930, cost 14.09 s 2023-09-04 18:22:07,186 44k INFO ====> Epoch: 3931, cost 14.17 s 2023-09-04 18:22:21,340 44k INFO ====> Epoch: 3932, cost 14.15 s 2023-09-04 18:22:35,368 44k INFO ====> Epoch: 3933, cost 14.03 s 2023-09-04 18:22:40,088 44k INFO Train Epoch: 3934 [29%] 2023-09-04 18:22:40,088 44k INFO Losses: [2.44235897064209, 2.636268138885498, 6.918532371520996, 14.612064361572266, 0.6003552079200745], step: 82600, lr: 6.116128900974286e-05, reference_loss: 27.209579467773438 2023-09-04 18:22:49,746 44k INFO ====> Epoch: 3934, cost 14.38 s 2023-09-04 18:23:03,985 44k INFO ====> Epoch: 3935, cost 14.24 s 2023-09-04 18:23:18,107 44k INFO ====> Epoch: 3936, cost 14.12 s 2023-09-04 18:23:32,055 44k INFO ====> Epoch: 3937, cost 13.95 s 2023-09-04 18:23:46,167 44k INFO ====> Epoch: 3938, cost 14.11 s 2023-09-04 18:24:00,577 44k INFO ====> Epoch: 3939, cost 14.41 s 2023-09-04 18:24:14,868 44k INFO ====> Epoch: 3940, cost 14.29 s 2023-09-04 18:24:29,045 44k INFO ====> Epoch: 3941, cost 14.18 s 2023-09-04 18:24:43,141 44k INFO ====> Epoch: 3942, cost 14.10 s 2023-09-04 18:24:55,733 44k INFO Train Epoch: 3943 [81%] 2023-09-04 18:24:55,733 44k INFO Losses: [2.2983222007751465, 2.4759268760681152, 7.053445339202881, 14.776300430297852, 0.5905896425247192], step: 82800, lr: 6.109251695279955e-05, reference_loss: 27.194583892822266 2023-09-04 18:24:57,790 44k INFO ====> Epoch: 3943, cost 14.65 s 2023-09-04 18:25:12,085 44k INFO ====> Epoch: 3944, cost 14.30 s 2023-09-04 18:25:26,158 44k INFO ====> Epoch: 3945, cost 14.07 s 2023-09-04 18:25:40,142 44k INFO ====> Epoch: 3946, cost 13.98 s 2023-09-04 18:25:54,351 44k INFO ====> Epoch: 3947, cost 14.21 s 2023-09-04 18:26:08,682 44k INFO ====> Epoch: 3948, cost 14.33 s 2023-09-04 18:26:22,817 44k INFO ====> Epoch: 3949, cost 14.13 s 2023-09-04 18:26:36,925 44k INFO ====> Epoch: 3950, cost 14.11 s 2023-09-04 18:26:51,082 44k INFO ====> Epoch: 3951, cost 14.16 s 2023-09-04 18:27:05,214 44k INFO ====> Epoch: 3952, cost 14.13 s 2023-09-04 18:27:10,741 44k INFO Train Epoch: 3953 [33%] 2023-09-04 18:27:10,741 44k INFO Losses: [2.2546048164367676, 2.556741714477539, 8.667657852172852, 16.04181671142578, 0.5729345679283142], step: 83000, lr: 6.101619424796909e-05, reference_loss: 30.0937557220459 2023-09-04 18:27:19,530 44k INFO ====> Epoch: 3953, cost 14.32 s 2023-09-04 18:27:33,546 44k INFO ====> Epoch: 3954, cost 14.02 s 2023-09-04 18:27:47,598 44k INFO ====> Epoch: 3955, cost 14.05 s 2023-09-04 18:28:01,837 44k INFO ====> Epoch: 3956, cost 14.24 s 2023-09-04 18:28:15,963 44k INFO ====> Epoch: 3957, cost 14.13 s 2023-09-04 18:28:30,034 44k INFO ====> Epoch: 3958, cost 14.07 s 2023-09-04 18:28:44,158 44k INFO ====> Epoch: 3959, cost 14.12 s 2023-09-04 18:28:58,528 44k INFO ====> Epoch: 3960, cost 14.37 s 2023-09-04 18:29:12,568 44k INFO ====> Epoch: 3961, cost 14.04 s 2023-09-04 18:29:25,571 44k INFO Train Epoch: 3962 [86%] 2023-09-04 18:29:25,572 44k INFO Losses: [2.1329612731933594, 2.8941826820373535, 8.632905006408691, 15.714872360229492, 0.6089188456535339], step: 83200, lr: 6.094758534104077e-05, reference_loss: 29.983840942382812 2023-09-04 18:29:30,286 44k INFO Saving model and optimizer state at iteration 3962 to ./logs\44k\G_83200.pth 2023-09-04 18:29:30,807 44k INFO Saving model and optimizer state at iteration 3962 to ./logs\44k\D_83200.pth 2023-09-04 18:29:32,595 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_80800.pth 2023-09-04 18:29:32,629 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_80800.pth 2023-09-04 18:29:33,761 44k INFO ====> Epoch: 3962, cost 21.19 s 2023-09-04 18:29:47,870 44k INFO ====> Epoch: 3963, cost 14.11 s 2023-09-04 18:30:02,393 44k INFO ====> Epoch: 3964, cost 14.52 s 2023-09-04 18:30:16,643 44k INFO ====> Epoch: 3965, cost 14.25 s 2023-09-04 18:30:30,853 44k INFO ====> Epoch: 3966, cost 14.21 s 2023-09-04 18:30:45,042 44k INFO ====> Epoch: 3967, cost 14.19 s 2023-09-04 18:30:59,511 44k INFO ====> Epoch: 3968, cost 14.47 s 2023-09-04 18:31:13,443 44k INFO ====> Epoch: 3969, cost 13.93 s 2023-09-04 18:31:27,334 44k INFO ====> Epoch: 3970, cost 13.89 s 2023-09-04 18:31:41,435 44k INFO ====> Epoch: 3971, cost 14.10 s 2023-09-04 18:31:47,628 44k INFO Train Epoch: 3972 [38%] 2023-09-04 18:31:47,628 44k INFO Losses: [2.2413718700408936, 2.833069324493408, 9.843905448913574, 16.647289276123047, 0.5781909227371216], step: 83400, lr: 6.087144369885392e-05, reference_loss: 32.14382553100586 2023-09-04 18:31:56,011 44k INFO ====> Epoch: 3972, cost 14.58 s 2023-09-04 18:32:10,272 44k INFO ====> Epoch: 3973, cost 14.26 s 2023-09-04 18:32:24,424 44k INFO ====> Epoch: 3974, cost 14.15 s 2023-09-04 18:32:38,474 44k INFO ====> Epoch: 3975, cost 14.05 s 2023-09-04 18:32:52,574 44k INFO ====> Epoch: 3976, cost 14.10 s 2023-09-04 18:33:06,783 44k INFO ====> Epoch: 3977, cost 14.21 s 2023-09-04 18:33:20,845 44k INFO ====> Epoch: 3978, cost 14.06 s 2023-09-04 18:33:34,721 44k INFO ====> Epoch: 3979, cost 13.88 s 2023-09-04 18:33:48,865 44k INFO ====> Epoch: 3980, cost 14.14 s 2023-09-04 18:34:02,653 44k INFO Train Epoch: 3981 [90%] 2023-09-04 18:34:02,654 44k INFO Losses: [2.110960006713867, 2.617983818054199, 7.006109714508057, 15.281209945678711, 0.48931458592414856], step: 83600, lr: 6.080299755489492e-05, reference_loss: 27.505578994750977 2023-09-04 18:34:03,335 44k INFO ====> Epoch: 3981, cost 14.47 s 2023-09-04 18:34:17,614 44k INFO ====> Epoch: 3982, cost 14.28 s 2023-09-04 18:34:31,647 44k INFO ====> Epoch: 3983, cost 14.03 s 2023-09-04 18:34:45,724 44k INFO ====> Epoch: 3984, cost 14.08 s 2023-09-04 18:35:00,010 44k INFO ====> Epoch: 3985, cost 14.29 s 2023-09-04 18:35:14,156 44k INFO ====> Epoch: 3986, cost 14.15 s 2023-09-04 18:35:27,995 44k INFO ====> Epoch: 3987, cost 13.84 s 2023-09-04 18:35:42,052 44k INFO ====> Epoch: 3988, cost 14.06 s 2023-09-04 18:35:56,426 44k INFO ====> Epoch: 3989, cost 14.37 s 2023-09-04 18:36:10,731 44k INFO ====> Epoch: 3990, cost 14.30 s 2023-09-04 18:36:17,736 44k INFO Train Epoch: 3991 [43%] 2023-09-04 18:36:17,736 44k INFO Losses: [2.342745065689087, 2.5732364654541016, 7.246225833892822, 14.645702362060547, 0.5297874212265015], step: 83800, lr: 6.0727036545811356e-05, reference_loss: 27.337696075439453 2023-09-04 18:36:25,355 44k INFO ====> Epoch: 3991, cost 14.62 s 2023-09-04 18:36:39,463 44k INFO ====> Epoch: 3992, cost 14.11 s 2023-09-04 18:36:53,700 44k INFO ====> Epoch: 3993, cost 14.24 s 2023-09-04 18:37:07,868 44k INFO ====> Epoch: 3994, cost 14.17 s 2023-09-04 18:37:21,921 44k INFO ====> Epoch: 3995, cost 14.05 s 2023-09-04 18:37:35,937 44k INFO ====> Epoch: 3996, cost 14.02 s 2023-09-04 18:37:50,131 44k INFO ====> Epoch: 3997, cost 14.19 s 2023-09-04 18:38:04,390 44k INFO ====> Epoch: 3998, cost 14.26 s 2023-09-04 18:38:18,544 44k INFO ====> Epoch: 3999, cost 14.15 s 2023-09-04 18:38:32,596 44k INFO Train Epoch: 4000 [95%] 2023-09-04 18:38:32,596 44k INFO Losses: [2.2415809631347656, 2.4584474563598633, 8.774938583374023, 16.283763885498047, 0.303534597158432], step: 84000, lr: 6.0658752778694185e-05, reference_loss: 30.062267303466797 2023-09-04 18:38:37,256 44k INFO Saving model and optimizer state at iteration 4000 to ./logs\44k\G_84000.pth 2023-09-04 18:38:37,827 44k INFO Saving model and optimizer state at iteration 4000 to ./logs\44k\D_84000.pth 2023-09-04 18:38:39,024 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_81600.pth 2023-09-04 18:38:39,059 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_81600.pth 2023-09-04 18:38:39,059 44k INFO ====> Epoch: 4000, cost 20.52 s 2023-09-04 18:38:53,412 44k INFO ====> Epoch: 4001, cost 14.35 s 2023-09-04 18:39:07,621 44k INFO ====> Epoch: 4002, cost 14.21 s 2023-09-04 18:39:21,604 44k INFO ====> Epoch: 4003, cost 13.98 s 2023-09-04 18:39:35,585 44k INFO ====> Epoch: 4004, cost 13.98 s 2023-09-04 18:39:49,622 44k INFO ====> Epoch: 4005, cost 14.04 s 2023-09-04 18:40:03,637 44k INFO ====> Epoch: 4006, cost 14.01 s 2023-09-04 18:40:17,803 44k INFO ====> Epoch: 4007, cost 14.17 s 2023-09-04 18:40:31,933 44k INFO ====> Epoch: 4008, cost 14.13 s 2023-09-04 18:40:46,044 44k INFO ====> Epoch: 4009, cost 14.11 s 2023-09-04 18:40:53,693 44k INFO Train Epoch: 4010 [48%] 2023-09-04 18:40:53,694 44k INFO Losses: [2.2182822227478027, 2.666721820831299, 8.580321311950684, 16.10875129699707, 0.7223567366600037], step: 84200, lr: 6.058297197419256e-05, reference_loss: 30.29643440246582 2023-09-04 18:41:00,603 44k INFO ====> Epoch: 4010, cost 14.56 s 2023-09-04 18:41:14,763 44k INFO ====> Epoch: 4011, cost 14.16 s 2023-09-04 18:41:28,712 44k INFO ====> Epoch: 4012, cost 13.95 s 2023-09-04 18:41:43,000 44k INFO ====> Epoch: 4013, cost 14.29 s 2023-09-04 18:41:57,481 44k INFO ====> Epoch: 4014, cost 14.48 s 2023-09-04 18:42:11,852 44k INFO ====> Epoch: 4015, cost 14.37 s 2023-09-04 18:42:25,914 44k INFO ====> Epoch: 4016, cost 14.06 s 2023-09-04 18:42:40,091 44k INFO ====> Epoch: 4017, cost 14.18 s 2023-09-04 18:42:54,521 44k INFO ====> Epoch: 4018, cost 14.43 s 2023-09-04 18:43:08,819 44k INFO ====> Epoch: 4019, cost 14.30 s 2023-09-04 18:43:09,518 44k INFO Train Epoch: 4020 [0%] 2023-09-04 18:43:09,518 44k INFO Losses: [2.344677209854126, 2.6327054500579834, 7.705936908721924, 14.167828559875488, 0.611531138420105], step: 84400, lr: 6.050728584243093e-05, reference_loss: 27.462677001953125 2023-09-04 18:43:23,018 44k INFO ====> Epoch: 4020, cost 14.20 s 2023-09-04 18:43:36,820 44k INFO ====> Epoch: 4021, cost 13.80 s 2023-09-04 18:43:51,077 44k INFO ====> Epoch: 4022, cost 14.26 s 2023-09-04 18:44:05,274 44k INFO ====> Epoch: 4023, cost 14.20 s 2023-09-04 18:44:19,493 44k INFO ====> Epoch: 4024, cost 14.22 s 2023-09-04 18:44:33,519 44k INFO ====> Epoch: 4025, cost 14.03 s 2023-09-04 18:44:47,737 44k INFO ====> Epoch: 4026, cost 14.22 s 2023-09-04 18:45:01,880 44k INFO ====> Epoch: 4027, cost 14.14 s 2023-09-04 18:45:16,009 44k INFO ====> Epoch: 4028, cost 14.13 s 2023-09-04 18:45:24,153 44k INFO Train Epoch: 4029 [52%] 2023-09-04 18:45:24,153 44k INFO Losses: [2.18603253364563, 2.8324060440063477, 8.579075813293457, 15.8204984664917, 0.4549389183521271], step: 84600, lr: 6.043924917128135e-05, reference_loss: 29.87295150756836 2023-09-04 18:45:30,310 44k INFO ====> Epoch: 4029, cost 14.30 s 2023-09-04 18:45:44,400 44k INFO ====> Epoch: 4030, cost 14.09 s 2023-09-04 18:45:58,467 44k INFO ====> Epoch: 4031, cost 14.07 s 2023-09-04 18:46:12,519 44k INFO ====> Epoch: 4032, cost 14.05 s 2023-09-04 18:46:26,532 44k INFO ====> Epoch: 4033, cost 14.01 s 2023-09-04 18:46:40,577 44k INFO ====> Epoch: 4034, cost 14.04 s 2023-09-04 18:46:55,017 44k INFO ====> Epoch: 4035, cost 14.44 s 2023-09-04 18:47:09,334 44k INFO ====> Epoch: 4036, cost 14.32 s 2023-09-04 18:47:23,299 44k INFO ====> Epoch: 4037, cost 13.96 s 2023-09-04 18:47:37,353 44k INFO ====> Epoch: 4038, cost 14.05 s 2023-09-04 18:47:38,720 44k INFO Train Epoch: 4039 [5%] 2023-09-04 18:47:38,720 44k INFO Losses: [2.245645523071289, 2.7796952724456787, 8.298649787902832, 15.502676963806152, 0.603167712688446], step: 84800, lr: 6.036374259200194e-05, reference_loss: 29.42983627319336 2023-09-04 18:47:43,414 44k INFO Saving model and optimizer state at iteration 4039 to ./logs\44k\G_84800.pth 2023-09-04 18:47:44,033 44k INFO Saving model and optimizer state at iteration 4039 to ./logs\44k\D_84800.pth 2023-09-04 18:47:45,198 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_82400.pth 2023-09-04 18:47:45,227 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_82400.pth 2023-09-04 18:47:58,063 44k INFO ====> Epoch: 4039, cost 20.71 s 2023-09-04 18:48:12,567 44k INFO ====> Epoch: 4040, cost 14.50 s 2023-09-04 18:48:26,587 44k INFO ====> Epoch: 4041, cost 14.02 s 2023-09-04 18:48:40,875 44k INFO ====> Epoch: 4042, cost 14.29 s 2023-09-04 18:48:55,310 44k INFO ====> Epoch: 4043, cost 14.44 s 2023-09-04 18:49:09,690 44k INFO ====> Epoch: 4044, cost 14.38 s 2023-09-04 18:49:23,720 44k INFO ====> Epoch: 4045, cost 14.03 s 2023-09-04 18:49:37,646 44k INFO ====> Epoch: 4046, cost 13.93 s 2023-09-04 18:49:51,904 44k INFO ====> Epoch: 4047, cost 14.26 s 2023-09-04 18:50:00,847 44k INFO Train Epoch: 4048 [57%] 2023-09-04 18:50:00,848 44k INFO Losses: [1.97903573513031, 3.297426700592041, 9.268476486206055, 15.33373737335205, 0.6088550090789795], step: 85000, lr: 6.0295867326289556e-05, reference_loss: 30.487531661987305 2023-09-04 18:50:06,387 44k INFO ====> Epoch: 4048, cost 14.48 s 2023-09-04 18:50:20,704 44k INFO ====> Epoch: 4049, cost 14.32 s 2023-09-04 18:50:34,845 44k INFO ====> Epoch: 4050, cost 14.14 s 2023-09-04 18:50:48,971 44k INFO ====> Epoch: 4051, cost 14.13 s 2023-09-04 18:51:03,148 44k INFO ====> Epoch: 4052, cost 14.18 s 2023-09-04 18:51:17,232 44k INFO ====> Epoch: 4053, cost 14.08 s 2023-09-04 18:51:31,117 44k INFO ====> Epoch: 4054, cost 13.89 s 2023-09-04 18:51:45,078 44k INFO ====> Epoch: 4055, cost 13.96 s 2023-09-04 18:51:59,166 44k INFO ====> Epoch: 4056, cost 14.09 s 2023-09-04 18:52:13,414 44k INFO ====> Epoch: 4057, cost 14.25 s 2023-09-04 18:52:15,486 44k INFO Train Epoch: 4058 [10%] 2023-09-04 18:52:15,486 44k INFO Losses: [2.4675798416137695, 2.5773088932037354, 7.1526618003845215, 15.426486015319824, 0.5595890283584595], step: 85200, lr: 6.022053987353462e-05, reference_loss: 28.183624267578125 2023-09-04 18:52:27,780 44k INFO ====> Epoch: 4058, cost 14.37 s 2023-09-04 18:52:41,862 44k INFO ====> Epoch: 4059, cost 14.08 s 2023-09-04 18:52:56,104 44k INFO ====> Epoch: 4060, cost 14.24 s 2023-09-04 18:53:10,217 44k INFO ====> Epoch: 4061, cost 14.11 s 2023-09-04 18:53:24,345 44k INFO ====> Epoch: 4062, cost 14.13 s 2023-09-04 18:53:38,246 44k INFO ====> Epoch: 4063, cost 13.90 s 2023-09-04 18:53:52,673 44k INFO ====> Epoch: 4064, cost 14.43 s 2023-09-04 18:54:06,950 44k INFO ====> Epoch: 4065, cost 14.28 s 2023-09-04 18:54:21,132 44k INFO ====> Epoch: 4066, cost 14.18 s 2023-09-04 18:54:30,765 44k INFO Train Epoch: 4067 [62%] 2023-09-04 18:54:30,765 44k INFO Losses: [2.288351535797119, 2.606593608856201, 8.556814193725586, 15.663403511047363, 0.5170607566833496], step: 85400, lr: 6.015282563035248e-05, reference_loss: 29.63222312927246 2023-09-04 18:54:35,509 44k INFO ====> Epoch: 4067, cost 14.38 s 2023-09-04 18:54:49,750 44k INFO ====> Epoch: 4068, cost 14.24 s 2023-09-04 18:55:04,108 44k INFO ====> Epoch: 4069, cost 14.36 s 2023-09-04 18:55:18,235 44k INFO ====> Epoch: 4070, cost 14.13 s 2023-09-04 18:55:32,157 44k INFO ====> Epoch: 4071, cost 13.92 s 2023-09-04 18:55:46,430 44k INFO ====> Epoch: 4072, cost 14.27 s 2023-09-04 18:56:00,627 44k INFO ====> Epoch: 4073, cost 14.20 s 2023-09-04 18:56:14,784 44k INFO ====> Epoch: 4074, cost 14.16 s 2023-09-04 18:56:28,869 44k INFO ====> Epoch: 4075, cost 14.08 s 2023-09-04 18:56:42,827 44k INFO ====> Epoch: 4076, cost 13.96 s 2023-09-04 18:56:45,582 44k INFO Train Epoch: 4077 [14%] 2023-09-04 18:56:45,583 44k INFO Losses: [2.205967426300049, 2.598809003829956, 9.908995628356934, 16.21560287475586, 0.6731414198875427], step: 85600, lr: 6.00776768791748e-05, reference_loss: 31.602516174316406 2023-09-04 18:56:50,275 44k INFO Saving model and optimizer state at iteration 4077 to ./logs\44k\G_85600.pth 2023-09-04 18:56:50,847 44k INFO Saving model and optimizer state at iteration 4077 to ./logs\44k\D_85600.pth 2023-09-04 18:56:52,434 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_83200.pth 2023-09-04 18:56:52,463 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_83200.pth 2023-09-04 18:57:03,983 44k INFO ====> Epoch: 4077, cost 21.16 s 2023-09-04 18:57:18,006 44k INFO ====> Epoch: 4078, cost 14.02 s 2023-09-04 18:57:31,851 44k INFO ====> Epoch: 4079, cost 13.84 s 2023-09-04 18:57:45,874 44k INFO ====> Epoch: 4080, cost 14.02 s 2023-09-04 18:57:59,984 44k INFO ====> Epoch: 4081, cost 14.11 s 2023-09-04 18:58:14,029 44k INFO ====> Epoch: 4082, cost 14.05 s 2023-09-04 18:58:28,165 44k INFO ====> Epoch: 4083, cost 14.14 s 2023-09-04 18:58:42,142 44k INFO ====> Epoch: 4084, cost 13.98 s 2023-09-04 18:58:56,435 44k INFO ====> Epoch: 4085, cost 14.29 s 2023-09-04 18:59:06,651 44k INFO Train Epoch: 4086 [67%] 2023-09-04 18:59:06,651 44k INFO Losses: [2.2781333923339844, 2.3962104320526123, 8.173572540283203, 15.995163917541504, 0.5436381697654724], step: 85800, lr: 6.001012327652431e-05, reference_loss: 29.38671875 2023-09-04 18:59:10,722 44k INFO ====> Epoch: 4086, cost 14.29 s 2023-09-04 18:59:24,656 44k INFO ====> Epoch: 4087, cost 13.93 s 2023-09-04 18:59:38,694 44k INFO ====> Epoch: 4088, cost 14.04 s 2023-09-04 18:59:53,021 44k INFO ====> Epoch: 4089, cost 14.33 s 2023-09-04 19:00:07,382 44k INFO ====> Epoch: 4090, cost 14.36 s 2023-09-04 19:00:21,644 44k INFO ====> Epoch: 4091, cost 14.26 s 2023-09-04 19:00:35,676 44k INFO ====> Epoch: 4092, cost 14.03 s 2023-09-04 19:00:49,763 44k INFO ====> Epoch: 4093, cost 14.09 s 2023-09-04 19:01:04,056 44k INFO ====> Epoch: 4094, cost 14.29 s 2023-09-04 19:01:18,274 44k INFO ====> Epoch: 4095, cost 14.22 s 2023-09-04 19:01:21,721 44k INFO Train Epoch: 4096 [19%] 2023-09-04 19:01:21,721 44k INFO Losses: [2.118896484375, 2.6546359062194824, 8.25195598602295, 15.93620491027832, 0.5122467875480652], step: 86000, lr: 5.993515280298476e-05, reference_loss: 29.473939895629883 2023-09-04 19:01:32,699 44k INFO ====> Epoch: 4096, cost 14.43 s 2023-09-04 19:01:46,826 44k INFO ====> Epoch: 4097, cost 14.13 s 2023-09-04 19:02:01,053 44k INFO ====> Epoch: 4098, cost 14.23 s 2023-09-04 19:02:15,298 44k INFO ====> Epoch: 4099, cost 14.25 s 2023-09-04 19:02:29,392 44k INFO ====> Epoch: 4100, cost 14.09 s 2023-09-04 19:02:43,464 44k INFO ====> Epoch: 4101, cost 14.07 s 2023-09-04 19:02:57,627 44k INFO ====> Epoch: 4102, cost 14.16 s 2023-09-04 19:03:11,855 44k INFO ====> Epoch: 4103, cost 14.23 s 2023-09-04 19:03:25,858 44k INFO ====> Epoch: 4104, cost 14.00 s 2023-09-04 19:03:36,747 44k INFO Train Epoch: 4105 [71%] 2023-09-04 19:03:36,748 44k INFO Losses: [2.477109432220459, 2.486490488052368, 7.725522994995117, 14.755751609802246, 0.4019584059715271], step: 86200, lr: 5.9867759459773554e-05, reference_loss: 27.846832275390625 2023-09-04 19:03:40,180 44k INFO ====> Epoch: 4105, cost 14.32 s 2023-09-04 19:03:54,089 44k INFO ====> Epoch: 4106, cost 13.91 s 2023-09-04 19:04:08,197 44k INFO ====> Epoch: 4107, cost 14.11 s 2023-09-04 19:04:22,246 44k INFO ====> Epoch: 4108, cost 14.05 s 2023-09-04 19:04:36,261 44k INFO ====> Epoch: 4109, cost 14.01 s 2023-09-04 19:04:50,579 44k INFO ====> Epoch: 4110, cost 14.32 s 2023-09-04 19:05:04,870 44k INFO ====> Epoch: 4111, cost 14.29 s 2023-09-04 19:05:18,859 44k INFO ====> Epoch: 4112, cost 13.99 s 2023-09-04 19:05:32,785 44k INFO ====> Epoch: 4113, cost 13.93 s 2023-09-04 19:05:47,007 44k INFO ====> Epoch: 4114, cost 14.22 s 2023-09-04 19:05:51,187 44k INFO Train Epoch: 4115 [24%] 2023-09-04 19:05:51,187 44k INFO Losses: [2.1430506706237793, 2.9350762367248535, 8.559443473815918, 15.151790618896484, 0.698213517665863], step: 86400, lr: 5.9792966840938744e-05, reference_loss: 29.487573623657227 2023-09-04 19:05:55,980 44k INFO Saving model and optimizer state at iteration 4115 to ./logs\44k\G_86400.pth 2023-09-04 19:05:56,710 44k INFO Saving model and optimizer state at iteration 4115 to ./logs\44k\D_86400.pth 2023-09-04 19:05:57,862 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_84000.pth 2023-09-04 19:05:57,897 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_84000.pth 2023-09-04 19:06:07,943 44k INFO ====> Epoch: 4115, cost 20.94 s 2023-09-04 19:06:21,949 44k INFO ====> Epoch: 4116, cost 14.01 s 2023-09-04 19:06:36,109 44k INFO ====> Epoch: 4117, cost 14.16 s 2023-09-04 19:06:50,340 44k INFO ====> Epoch: 4118, cost 14.23 s 2023-09-04 19:07:04,653 44k INFO ====> Epoch: 4119, cost 14.31 s 2023-09-04 19:07:18,718 44k INFO ====> Epoch: 4120, cost 14.07 s 2023-09-04 19:07:32,581 44k INFO ====> Epoch: 4121, cost 13.86 s 2023-09-04 19:07:46,735 44k INFO ====> Epoch: 4122, cost 14.15 s 2023-09-04 19:08:01,182 44k INFO ====> Epoch: 4123, cost 14.45 s 2023-09-04 19:08:12,841 44k INFO Train Epoch: 4124 [76%] 2023-09-04 19:08:12,841 44k INFO Losses: [2.2713918685913086, 2.4066848754882812, 8.545063018798828, 16.436012268066406, 0.7070581912994385], step: 86600, lr: 5.9725733376978566e-05, reference_loss: 30.3662109375 2023-09-04 19:08:15,600 44k INFO ====> Epoch: 4124, cost 14.42 s 2023-09-04 19:08:29,532 44k INFO ====> Epoch: 4125, cost 13.93 s 2023-09-04 19:08:43,548 44k INFO ====> Epoch: 4126, cost 14.02 s 2023-09-04 19:08:57,698 44k INFO ====> Epoch: 4127, cost 14.15 s 2023-09-04 19:09:12,063 44k INFO ====> Epoch: 4128, cost 14.37 s 2023-09-04 19:09:26,009 44k INFO ====> Epoch: 4129, cost 13.95 s 2023-09-04 19:09:40,025 44k INFO ====> Epoch: 4130, cost 14.02 s 2023-09-04 19:09:54,290 44k INFO ====> Epoch: 4131, cost 14.27 s 2023-09-04 19:10:08,529 44k INFO ====> Epoch: 4132, cost 14.24 s 2023-09-04 19:10:22,527 44k INFO ====> Epoch: 4133, cost 14.00 s 2023-09-04 19:10:27,341 44k INFO Train Epoch: 4134 [29%] 2023-09-04 19:10:27,342 44k INFO Losses: [2.3432774543762207, 2.417898654937744, 8.047013282775879, 16.12914276123047, 0.5357145667076111], step: 86800, lr: 5.9651118190918446e-05, reference_loss: 29.473047256469727 2023-09-04 19:10:36,977 44k INFO ====> Epoch: 4134, cost 14.45 s 2023-09-04 19:10:51,035 44k INFO ====> Epoch: 4135, cost 14.06 s 2023-09-04 19:11:05,310 44k INFO ====> Epoch: 4136, cost 14.27 s 2023-09-04 19:11:19,410 44k INFO ====> Epoch: 4137, cost 14.10 s 2023-09-04 19:11:33,387 44k INFO ====> Epoch: 4138, cost 13.98 s 2023-09-04 19:11:47,500 44k INFO ====> Epoch: 4139, cost 14.11 s 2023-09-04 19:12:01,959 44k INFO ====> Epoch: 4140, cost 14.46 s 2023-09-04 19:12:16,185 44k INFO ====> Epoch: 4141, cost 14.23 s 2023-09-04 19:12:30,331 44k INFO ====> Epoch: 4142, cost 14.15 s 2023-09-04 19:12:42,772 44k INFO Train Epoch: 4143 [81%] 2023-09-04 19:12:42,772 44k INFO Losses: [2.177560329437256, 2.640416383743286, 8.864034652709961, 15.000662803649902, 0.7006136178970337], step: 87000, lr: 5.958404422692294e-05, reference_loss: 29.38328742980957 2023-09-04 19:12:44,786 44k INFO ====> Epoch: 4143, cost 14.45 s 2023-09-04 19:12:59,218 44k INFO ====> Epoch: 4144, cost 14.43 s 2023-09-04 19:13:13,535 44k INFO ====> Epoch: 4145, cost 14.32 s 2023-09-04 19:13:27,571 44k INFO ====> Epoch: 4146, cost 14.04 s 2023-09-04 19:13:41,512 44k INFO ====> Epoch: 4147, cost 13.94 s 2023-09-04 19:13:55,687 44k INFO ====> Epoch: 4148, cost 14.17 s 2023-09-04 19:14:09,980 44k INFO ====> Epoch: 4149, cost 14.29 s 2023-09-04 19:14:24,166 44k INFO ====> Epoch: 4150, cost 14.19 s 2023-09-04 19:14:38,259 44k INFO ====> Epoch: 4151, cost 14.09 s 2023-09-04 19:14:52,234 44k INFO ====> Epoch: 4152, cost 13.98 s 2023-09-04 19:14:57,763 44k INFO Train Epoch: 4153 [33%] 2023-09-04 19:14:57,763 44k INFO Losses: [2.5353305339813232, 2.267490863800049, 5.881820201873779, 14.903647422790527, 0.6150005459785461], step: 87200, lr: 5.95096060527084e-05, reference_loss: 26.20328712463379 2023-09-04 19:15:02,504 44k INFO Saving model and optimizer state at iteration 4153 to ./logs\44k\G_87200.pth 2023-09-04 19:15:03,162 44k INFO Saving model and optimizer state at iteration 4153 to ./logs\44k\D_87200.pth 2023-09-04 19:15:04,244 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_84800.pth 2023-09-04 19:15:04,279 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_84800.pth 2023-09-04 19:15:12,953 44k INFO ====> Epoch: 4153, cost 20.72 s 2023-09-04 19:15:26,899 44k INFO ====> Epoch: 4154, cost 13.94 s 2023-09-04 19:15:40,855 44k INFO ====> Epoch: 4155, cost 13.96 s 2023-09-04 19:15:54,966 44k INFO ====> Epoch: 4156, cost 14.11 s 2023-09-04 19:16:09,131 44k INFO ====> Epoch: 4157, cost 14.16 s 2023-09-04 19:16:23,176 44k INFO ====> Epoch: 4158, cost 14.05 s 2023-09-04 19:16:37,151 44k INFO ====> Epoch: 4159, cost 13.97 s 2023-09-04 19:16:51,278 44k INFO ====> Epoch: 4160, cost 14.13 s 2023-09-04 19:17:05,654 44k INFO ====> Epoch: 4161, cost 14.38 s 2023-09-04 19:17:18,631 44k INFO Train Epoch: 4162 [86%] 2023-09-04 19:17:18,632 44k INFO Losses: [2.2601261138916016, 2.843902587890625, 8.228314399719238, 15.39985466003418, 0.7068935036659241], step: 87400, lr: 5.944269121029102e-05, reference_loss: 29.439090728759766 2023-09-04 19:17:19,977 44k INFO ====> Epoch: 4162, cost 14.32 s 2023-09-04 19:17:33,973 44k INFO ====> Epoch: 4163, cost 14.00 s 2023-09-04 19:17:48,088 44k INFO ====> Epoch: 4164, cost 14.12 s 2023-09-04 19:18:02,419 44k INFO ====> Epoch: 4165, cost 14.33 s 2023-09-04 19:18:16,645 44k INFO ====> Epoch: 4166, cost 14.23 s 2023-09-04 19:18:30,791 44k INFO ====> Epoch: 4167, cost 14.15 s 2023-09-04 19:18:45,000 44k INFO ====> Epoch: 4168, cost 14.21 s 2023-09-04 19:18:59,339 44k INFO ====> Epoch: 4169, cost 14.34 s 2023-09-04 19:19:13,467 44k INFO ====> Epoch: 4170, cost 14.13 s 2023-09-04 19:19:27,492 44k INFO ====> Epoch: 4171, cost 14.03 s 2023-09-04 19:19:33,602 44k INFO Train Epoch: 4172 [38%] 2023-09-04 19:19:33,602 44k INFO Losses: [2.4778292179107666, 2.173442840576172, 6.074784755706787, 15.012375831604004, 0.5723099112510681], step: 87600, lr: 5.936842962799156e-05, reference_loss: 26.31074333190918 2023-09-04 19:19:41,900 44k INFO ====> Epoch: 4172, cost 14.41 s 2023-09-04 19:19:56,072 44k INFO ====> Epoch: 4173, cost 14.17 s 2023-09-04 19:20:10,275 44k INFO ====> Epoch: 4174, cost 14.20 s 2023-09-04 19:20:24,365 44k INFO ====> Epoch: 4175, cost 14.09 s 2023-09-04 19:20:38,407 44k INFO ====> Epoch: 4176, cost 14.04 s 2023-09-04 19:20:52,599 44k INFO ====> Epoch: 4177, cost 14.19 s 2023-09-04 19:21:06,835 44k INFO ====> Epoch: 4178, cost 14.24 s 2023-09-04 19:21:20,878 44k INFO ====> Epoch: 4179, cost 14.04 s 2023-09-04 19:21:34,747 44k INFO ====> Epoch: 4180, cost 13.87 s 2023-09-04 19:21:48,243 44k INFO Train Epoch: 4181 [90%] 2023-09-04 19:21:48,243 44k INFO Losses: [2.049506664276123, 2.827176570892334, 9.923498153686523, 16.545621871948242, 0.42417916655540466], step: 87800, lr: 5.9301673529663405e-05, reference_loss: 31.769981384277344 2023-09-04 19:21:48,905 44k INFO ====> Epoch: 4181, cost 14.16 s 2023-09-04 19:22:03,155 44k INFO ====> Epoch: 4182, cost 14.25 s 2023-09-04 19:22:17,273 44k INFO ====> Epoch: 4183, cost 14.12 s 2023-09-04 19:22:31,219 44k INFO ====> Epoch: 4184, cost 13.95 s 2023-09-04 19:22:45,256 44k INFO ====> Epoch: 4185, cost 14.04 s 2023-09-04 19:22:59,643 44k INFO ====> Epoch: 4186, cost 14.39 s 2023-09-04 19:23:13,870 44k INFO ====> Epoch: 4187, cost 14.23 s 2023-09-04 19:23:27,799 44k INFO ====> Epoch: 4188, cost 13.93 s 2023-09-04 19:23:41,812 44k INFO ====> Epoch: 4189, cost 14.01 s 2023-09-04 19:23:56,271 44k INFO ====> Epoch: 4190, cost 14.46 s 2023-09-04 19:24:03,246 44k INFO Train Epoch: 4191 [43%] 2023-09-04 19:24:03,246 44k INFO Losses: [2.5296401977539062, 2.3958687782287598, 6.957724094390869, 14.081343650817871, 0.6764259934425354], step: 88000, lr: 5.922758812034471e-05, reference_loss: 26.641002655029297 2023-09-04 19:24:07,909 44k INFO Saving model and optimizer state at iteration 4191 to ./logs\44k\G_88000.pth 2023-09-04 19:24:08,603 44k INFO Saving model and optimizer state at iteration 4191 to ./logs\44k\D_88000.pth 2023-09-04 19:24:09,680 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_85600.pth 2023-09-04 19:24:09,710 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_85600.pth 2023-09-04 19:24:17,037 44k INFO ====> Epoch: 4191, cost 20.77 s 2023-09-04 19:24:31,058 44k INFO ====> Epoch: 4192, cost 14.02 s 2023-09-04 19:24:45,139 44k INFO ====> Epoch: 4193, cost 14.08 s 2023-09-04 19:24:59,392 44k INFO ====> Epoch: 4194, cost 14.25 s 2023-09-04 19:25:13,878 44k INFO ====> Epoch: 4195, cost 14.49 s 2023-09-04 19:25:27,973 44k INFO ====> Epoch: 4196, cost 14.09 s 2023-09-04 19:25:42,340 44k INFO ====> Epoch: 4197, cost 14.37 s 2023-09-04 19:25:57,083 44k INFO ====> Epoch: 4198, cost 14.74 s 2023-09-04 19:26:11,703 44k INFO ====> Epoch: 4199, cost 14.62 s 2023-09-04 19:26:26,258 44k INFO Train Epoch: 4200 [95%] 2023-09-04 19:26:26,258 44k INFO Losses: [2.39225435256958, 2.555236577987671, 7.108771324157715, 14.854989051818848, 0.5660098791122437], step: 88200, lr: 5.916099038951241e-05, reference_loss: 27.47726058959961 2023-09-04 19:26:26,600 44k INFO ====> Epoch: 4200, cost 14.90 s 2023-09-04 19:26:41,036 44k INFO ====> Epoch: 4201, cost 14.44 s 2023-09-04 19:26:55,551 44k INFO ====> Epoch: 4202, cost 14.52 s 2023-09-04 19:27:10,140 44k INFO ====> Epoch: 4203, cost 14.59 s 2023-09-04 19:27:24,262 44k INFO ====> Epoch: 4204, cost 14.12 s 2023-09-04 19:27:38,522 44k INFO ====> Epoch: 4205, cost 14.26 s 2023-09-04 19:27:53,028 44k INFO ====> Epoch: 4206, cost 14.51 s 2023-09-04 19:28:07,533 44k INFO ====> Epoch: 4207, cost 14.50 s 2023-09-04 19:28:21,803 44k INFO ====> Epoch: 4208, cost 14.27 s 2023-09-04 19:28:36,178 44k INFO ====> Epoch: 4209, cost 14.38 s 2023-09-04 19:28:44,043 44k INFO Train Epoch: 4210 [48%] 2023-09-04 19:28:44,043 44k INFO Losses: [2.390824794769287, 2.4736905097961426, 9.846138000488281, 17.597904205322266, 0.6284107565879822], step: 88400, lr: 5.908708073523403e-05, reference_loss: 32.93696594238281 2023-09-04 19:28:51,254 44k INFO ====> Epoch: 4210, cost 15.08 s 2023-09-04 19:29:05,875 44k INFO ====> Epoch: 4211, cost 14.62 s 2023-09-04 19:29:20,351 44k INFO ====> Epoch: 4212, cost 14.48 s 2023-09-04 19:29:34,738 44k INFO ====> Epoch: 4213, cost 14.39 s 2023-09-04 19:29:49,608 44k INFO ====> Epoch: 4214, cost 14.87 s 2023-09-04 19:30:04,507 44k INFO ====> Epoch: 4215, cost 14.90 s 2023-09-04 19:30:19,254 44k INFO ====> Epoch: 4216, cost 14.75 s 2023-09-04 19:30:33,601 44k INFO ====> Epoch: 4217, cost 14.35 s 2023-09-04 19:30:48,075 44k INFO ====> Epoch: 4218, cost 14.47 s 2023-09-04 19:31:02,986 44k INFO ====> Epoch: 4219, cost 14.91 s 2023-09-04 19:31:03,720 44k INFO Train Epoch: 4220 [0%] 2023-09-04 19:31:03,720 44k INFO Losses: [2.4511618614196777, 2.5333452224731445, 7.462044715881348, 14.486023902893066, 0.6758987903594971], step: 88600, lr: 5.901326341607309e-05, reference_loss: 27.608474731445312 2023-09-04 19:31:17,808 44k INFO ====> Epoch: 4220, cost 14.82 s 2023-09-04 19:31:32,104 44k INFO ====> Epoch: 4221, cost 14.30 s 2023-09-04 19:31:46,768 44k INFO ====> Epoch: 4222, cost 14.66 s 2023-09-04 19:32:01,355 44k INFO ====> Epoch: 4223, cost 14.59 s 2023-09-04 19:32:16,021 44k INFO ====> Epoch: 4224, cost 14.67 s 2023-09-04 19:32:30,408 44k INFO ====> Epoch: 4225, cost 14.39 s 2023-09-04 19:32:44,933 44k INFO ====> Epoch: 4226, cost 14.52 s 2023-09-04 19:32:59,581 44k INFO ====> Epoch: 4227, cost 14.65 s 2023-09-04 19:33:14,124 44k INFO ====> Epoch: 4228, cost 14.54 s 2023-09-04 19:33:22,382 44k INFO Train Epoch: 4229 [52%] 2023-09-04 19:33:22,383 44k INFO Losses: [2.0741515159606934, 2.981689214706421, 10.263463973999023, 16.86744499206543, 0.6272304058074951], step: 88800, lr: 5.8946906680010626e-05, reference_loss: 32.81398010253906 2023-09-04 19:33:27,733 44k INFO Saving model and optimizer state at iteration 4229 to ./logs\44k\G_88800.pth 2023-09-04 19:33:28,369 44k INFO Saving model and optimizer state at iteration 4229 to ./logs\44k\D_88800.pth 2023-09-04 19:33:29,445 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_86400.pth 2023-09-04 19:33:29,478 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_86400.pth 2023-09-04 19:33:35,459 44k INFO ====> Epoch: 4229, cost 21.33 s 2023-09-04 19:33:50,017 44k INFO ====> Epoch: 4230, cost 14.56 s 2023-09-04 19:34:04,604 44k INFO ====> Epoch: 4231, cost 14.59 s 2023-09-04 19:34:19,037 44k INFO ====> Epoch: 4232, cost 14.43 s 2023-09-04 19:34:33,546 44k INFO ====> Epoch: 4233, cost 14.51 s 2023-09-04 19:34:48,038 44k INFO ====> Epoch: 4234, cost 14.49 s 2023-09-04 19:35:02,622 44k INFO ====> Epoch: 4235, cost 14.58 s 2023-09-04 19:35:17,097 44k INFO ====> Epoch: 4236, cost 14.48 s 2023-09-04 19:35:31,429 44k INFO ====> Epoch: 4237, cost 14.33 s 2023-09-04 19:35:46,033 44k INFO ====> Epoch: 4238, cost 14.60 s 2023-09-04 19:35:47,515 44k INFO Train Epoch: 4239 [5%] 2023-09-04 19:35:47,516 44k INFO Losses: [2.421393394470215, 2.7010416984558105, 6.784772872924805, 13.860191345214844, 0.6399994492530823], step: 89000, lr: 5.887326447989169e-05, reference_loss: 26.407398223876953 2023-09-04 19:36:01,277 44k INFO ====> Epoch: 4239, cost 15.24 s 2023-09-04 19:36:15,944 44k INFO ====> Epoch: 4240, cost 14.67 s 2023-09-04 19:36:30,276 44k INFO ====> Epoch: 4241, cost 14.33 s 2023-09-04 19:36:45,029 44k INFO ====> Epoch: 4242, cost 14.75 s 2023-09-04 19:36:59,808 44k INFO ====> Epoch: 4243, cost 14.78 s 2023-09-04 19:37:14,469 44k INFO ====> Epoch: 4244, cost 14.66 s 2023-09-04 19:37:28,805 44k INFO ====> Epoch: 4245, cost 14.34 s 2023-09-04 19:37:43,067 44k INFO ====> Epoch: 4246, cost 14.26 s 2023-09-04 19:37:57,884 44k INFO ====> Epoch: 4247, cost 14.82 s 2023-09-04 19:38:07,187 44k INFO Train Epoch: 4248 [57%] 2023-09-04 19:38:07,188 44k INFO Losses: [2.2779600620269775, 2.5718495845794678, 7.426496982574463, 16.088897705078125, 0.5298312306404114], step: 89200, lr: 5.880706516390598e-05, reference_loss: 28.895034790039062 2023-09-04 19:38:12,843 44k INFO ====> Epoch: 4248, cost 14.96 s 2023-09-04 19:38:27,154 44k INFO ====> Epoch: 4249, cost 14.31 s 2023-09-04 19:38:41,488 44k INFO ====> Epoch: 4250, cost 14.33 s 2023-09-04 19:38:56,075 44k INFO ====> Epoch: 4251, cost 14.59 s 2023-09-04 19:39:10,735 44k INFO ====> Epoch: 4252, cost 14.66 s 2023-09-04 19:39:24,847 44k INFO ====> Epoch: 4253, cost 14.11 s 2023-09-04 19:39:39,109 44k INFO ====> Epoch: 4254, cost 14.26 s 2023-09-04 19:39:53,496 44k INFO ====> Epoch: 4255, cost 14.39 s 2023-09-04 19:40:08,046 44k INFO ====> Epoch: 4256, cost 14.55 s 2023-09-04 19:40:22,435 44k INFO ====> Epoch: 4257, cost 14.39 s 2023-09-04 19:40:24,531 44k INFO Train Epoch: 4258 [10%] 2023-09-04 19:40:24,531 44k INFO Losses: [2.3790740966796875, 2.535264253616333, 6.786473751068115, 14.111892700195312, 0.6481488347053528], step: 89400, lr: 5.873359766738887e-05, reference_loss: 26.460853576660156 2023-09-04 19:40:36,942 44k INFO ====> Epoch: 4258, cost 14.51 s 2023-09-04 19:40:51,478 44k INFO ====> Epoch: 4259, cost 14.54 s 2023-09-04 19:41:06,185 44k INFO ====> Epoch: 4260, cost 14.71 s 2023-09-04 19:41:20,769 44k INFO ====> Epoch: 4261, cost 14.58 s 2023-09-04 19:41:35,045 44k INFO ====> Epoch: 4262, cost 14.28 s 2023-09-04 19:41:49,779 44k INFO ====> Epoch: 4263, cost 14.73 s 2023-09-04 19:42:04,778 44k INFO ====> Epoch: 4264, cost 15.00 s 2023-09-04 19:42:19,335 44k INFO ====> Epoch: 4265, cost 14.56 s 2023-09-04 19:42:33,766 44k INFO ====> Epoch: 4266, cost 14.43 s 2023-09-04 19:42:43,758 44k INFO Train Epoch: 4267 [62%] 2023-09-04 19:42:43,759 44k INFO Losses: [2.526193857192993, 2.1893982887268066, 7.243322372436523, 14.587896347045898, 0.5798040628433228], step: 89600, lr: 5.866755539802755e-05, reference_loss: 27.12661361694336 2023-09-04 19:42:49,119 44k INFO Saving model and optimizer state at iteration 4267 to ./logs\44k\G_89600.pth 2023-09-04 19:42:49,725 44k INFO Saving model and optimizer state at iteration 4267 to ./logs\44k\D_89600.pth 2023-09-04 19:42:51,028 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_87200.pth 2023-09-04 19:42:51,063 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_87200.pth 2023-09-04 19:42:55,837 44k INFO ====> Epoch: 4267, cost 22.07 s 2023-09-04 19:43:10,434 44k INFO ====> Epoch: 4268, cost 14.60 s 2023-09-04 19:43:24,632 44k INFO ====> Epoch: 4269, cost 14.20 s 2023-09-04 19:43:39,014 44k INFO ====> Epoch: 4270, cost 14.38 s 2023-09-04 19:43:53,749 44k INFO ====> Epoch: 4271, cost 14.74 s 2023-09-04 19:44:08,508 44k INFO ====> Epoch: 4272, cost 14.76 s 2023-09-04 19:44:22,872 44k INFO ====> Epoch: 4273, cost 14.36 s 2023-09-04 19:44:37,363 44k INFO ====> Epoch: 4274, cost 14.49 s 2023-09-04 19:44:51,899 44k INFO ====> Epoch: 4275, cost 14.54 s 2023-09-04 19:45:06,568 44k INFO ====> Epoch: 4276, cost 14.67 s 2023-09-04 19:45:09,434 44k INFO Train Epoch: 4277 [14%] 2023-09-04 19:45:09,434 44k INFO Losses: [2.305663585662842, 2.5184803009033203, 10.150569915771484, 17.63759422302246, 0.7111982703208923], step: 89800, lr: 5.859426219065768e-05, reference_loss: 33.32350540161133 2023-09-04 19:45:21,204 44k INFO ====> Epoch: 4277, cost 14.64 s 2023-09-04 19:45:35,483 44k INFO ====> Epoch: 4278, cost 14.28 s 2023-09-04 19:45:49,923 44k INFO ====> Epoch: 4279, cost 14.44 s 2023-09-04 19:46:04,520 44k INFO ====> Epoch: 4280, cost 14.60 s 2023-09-04 19:46:18,878 44k INFO ====> Epoch: 4281, cost 14.36 s 2023-09-04 19:46:33,213 44k INFO ====> Epoch: 4282, cost 14.34 s 2023-09-04 19:46:47,789 44k INFO ====> Epoch: 4283, cost 14.58 s 2023-09-04 19:47:02,306 44k INFO ====> Epoch: 4284, cost 14.52 s 2023-09-04 19:47:16,756 44k INFO ====> Epoch: 4285, cost 14.45 s 2023-09-04 19:47:27,163 44k INFO Train Epoch: 4286 [67%] 2023-09-04 19:47:27,163 44k INFO Losses: [2.3011600971221924, 2.6439146995544434, 9.806249618530273, 16.341045379638672, 0.5481312274932861], step: 90000, lr: 5.852837659535434e-05, reference_loss: 31.640501022338867 2023-09-04 19:47:31,281 44k INFO ====> Epoch: 4286, cost 14.53 s 2023-09-04 19:47:45,808 44k INFO ====> Epoch: 4287, cost 14.53 s 2023-09-04 19:48:00,641 44k INFO ====> Epoch: 4288, cost 14.83 s 2023-09-04 19:48:15,396 44k INFO ====> Epoch: 4289, cost 14.76 s 2023-09-04 19:48:29,969 44k INFO ====> Epoch: 4290, cost 14.57 s 2023-09-04 19:48:44,606 44k INFO ====> Epoch: 4291, cost 14.64 s 2023-09-04 19:48:59,232 44k INFO ====> Epoch: 4292, cost 14.63 s 2023-09-04 19:49:13,797 44k INFO ====> Epoch: 4293, cost 14.57 s 2023-09-04 19:49:28,091 44k INFO ====> Epoch: 4294, cost 14.29 s 2023-09-04 19:49:42,539 44k INFO ====> Epoch: 4295, cost 14.45 s 2023-09-04 19:49:46,085 44k INFO Train Epoch: 4296 [19%] 2023-09-04 19:49:46,086 44k INFO Losses: [2.2065067291259766, 2.6547281742095947, 9.1409273147583, 15.797066688537598, 0.5280018448829651], step: 90200, lr: 5.845525726366033e-05, reference_loss: 30.32723045349121 2023-09-04 19:49:57,461 44k INFO ====> Epoch: 4296, cost 14.92 s 2023-09-04 19:50:12,144 44k INFO ====> Epoch: 4297, cost 14.68 s 2023-09-04 19:50:26,609 44k INFO ====> Epoch: 4298, cost 14.47 s 2023-09-04 19:50:40,987 44k INFO ====> Epoch: 4299, cost 14.38 s 2023-09-04 19:50:55,611 44k INFO ====> Epoch: 4300, cost 14.62 s 2023-09-04 19:51:10,052 44k INFO ====> Epoch: 4301, cost 14.44 s 2023-09-04 19:51:24,283 44k INFO ====> Epoch: 4302, cost 14.23 s 2023-09-04 19:51:38,738 44k INFO ====> Epoch: 4303, cost 14.46 s 2023-09-04 19:51:53,273 44k INFO ====> Epoch: 4304, cost 14.53 s 2023-09-04 19:52:04,563 44k INFO Train Epoch: 4305 [71%] 2023-09-04 19:52:04,563 44k INFO Losses: [2.1923811435699463, 2.509571075439453, 7.849832534790039, 15.297544479370117, 0.5979578495025635], step: 90400, lr: 5.838952797073237e-05, reference_loss: 28.44728660583496 2023-09-04 19:52:09,955 44k INFO Saving model and optimizer state at iteration 4305 to ./logs\44k\G_90400.pth 2023-09-04 19:52:10,608 44k INFO Saving model and optimizer state at iteration 4305 to ./logs\44k\D_90400.pth 2023-09-04 19:52:11,735 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_88000.pth 2023-09-04 19:52:11,768 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_88000.pth 2023-09-04 19:52:15,059 44k INFO ====> Epoch: 4305, cost 21.79 s 2023-09-04 19:52:29,566 44k INFO ====> Epoch: 4306, cost 14.51 s 2023-09-04 19:52:43,931 44k INFO ====> Epoch: 4307, cost 14.36 s 2023-09-04 19:52:58,589 44k INFO ====> Epoch: 4308, cost 14.66 s 2023-09-04 19:53:13,181 44k INFO ====> Epoch: 4309, cost 14.59 s 2023-09-04 19:53:27,444 44k INFO ====> Epoch: 4310, cost 14.26 s 2023-09-04 19:53:41,811 44k INFO ====> Epoch: 4311, cost 14.37 s 2023-09-04 19:53:56,708 44k INFO ====> Epoch: 4312, cost 14.90 s 2023-09-04 19:54:11,562 44k INFO ====> Epoch: 4313, cost 14.85 s 2023-09-04 19:54:25,955 44k INFO ====> Epoch: 4314, cost 14.39 s 2023-09-04 19:54:30,123 44k INFO Train Epoch: 4315 [24%] 2023-09-04 19:54:30,123 44k INFO Losses: [2.4761509895324707, 2.294882297515869, 6.566839694976807, 14.286782264709473, 0.6147669553756714], step: 90600, lr: 5.8316582102223745e-05, reference_loss: 26.239423751831055 2023-09-04 19:54:40,803 44k INFO ====> Epoch: 4315, cost 14.85 s 2023-09-04 19:54:55,524 44k INFO ====> Epoch: 4316, cost 14.72 s 2023-09-04 19:55:10,336 44k INFO ====> Epoch: 4317, cost 14.81 s 2023-09-04 19:55:24,536 44k INFO ====> Epoch: 4318, cost 14.20 s 2023-09-04 19:55:38,732 44k INFO ====> Epoch: 4319, cost 14.20 s 2023-09-04 19:55:53,198 44k INFO ====> Epoch: 4320, cost 14.47 s 2023-09-04 19:56:07,961 44k INFO ====> Epoch: 4321, cost 14.76 s 2023-09-04 19:56:22,437 44k INFO ====> Epoch: 4322, cost 14.48 s 2023-09-04 19:56:36,995 44k INFO ====> Epoch: 4323, cost 14.56 s 2023-09-04 19:56:49,072 44k INFO Train Epoch: 4324 [76%] 2023-09-04 19:56:49,073 44k INFO Losses: [2.2722902297973633, 2.592254638671875, 8.745088577270508, 16.83167839050293, 0.5031494498252869], step: 90800, lr: 5.8251008740870375e-05, reference_loss: 30.944459915161133 2023-09-04 19:56:51,890 44k INFO ====> Epoch: 4324, cost 14.90 s 2023-09-04 19:57:06,441 44k INFO ====> Epoch: 4325, cost 14.55 s 2023-09-04 19:57:20,879 44k INFO ====> Epoch: 4326, cost 14.44 s 2023-09-04 19:57:35,157 44k INFO ====> Epoch: 4327, cost 14.28 s 2023-09-04 19:57:49,692 44k INFO ====> Epoch: 4328, cost 14.54 s 2023-09-04 19:58:04,319 44k INFO ====> Epoch: 4329, cost 14.63 s 2023-09-04 19:58:18,755 44k INFO ====> Epoch: 4330, cost 14.44 s 2023-09-04 19:58:33,261 44k INFO ====> Epoch: 4331, cost 14.51 s 2023-09-04 19:58:47,609 44k INFO ====> Epoch: 4332, cost 14.35 s 2023-09-04 19:59:02,239 44k INFO ====> Epoch: 4333, cost 14.63 s 2023-09-04 19:59:07,212 44k INFO Train Epoch: 4334 [29%] 2023-09-04 19:59:07,212 44k INFO Losses: [2.0034844875335693, 2.9530694484710693, 8.512550354003906, 13.81867504119873, 0.5144368410110474], step: 91000, lr: 5.817823592403519e-05, reference_loss: 27.802217483520508 2023-09-04 19:59:16,874 44k INFO ====> Epoch: 4334, cost 14.64 s 2023-09-04 19:59:31,166 44k INFO ====> Epoch: 4335, cost 14.29 s 2023-09-04 19:59:45,683 44k INFO ====> Epoch: 4336, cost 14.52 s 2023-09-04 20:00:00,678 44k INFO ====> Epoch: 4337, cost 15.00 s 2023-09-04 20:00:15,330 44k INFO ====> Epoch: 4338, cost 14.65 s 2023-09-04 20:00:29,695 44k INFO ====> Epoch: 4339, cost 14.37 s 2023-09-04 20:00:44,163 44k INFO ====> Epoch: 4340, cost 14.47 s 2023-09-04 20:00:58,868 44k INFO ====> Epoch: 4341, cost 14.70 s 2023-09-04 20:01:13,578 44k INFO ====> Epoch: 4342, cost 14.71 s 2023-09-04 20:01:26,072 44k INFO Train Epoch: 4343 [81%] 2023-09-04 20:01:26,072 44k INFO Losses: [2.418954610824585, 2.4813780784606934, 8.253942489624023, 14.589004516601562, 0.6085371971130371], step: 91200, lr: 5.811281812433527e-05, reference_loss: 28.351816177368164 2023-09-04 20:01:31,460 44k INFO Saving model and optimizer state at iteration 4343 to ./logs\44k\G_91200.pth 2023-09-04 20:01:32,090 44k INFO Saving model and optimizer state at iteration 4343 to ./logs\44k\D_91200.pth 2023-09-04 20:01:33,259 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_88800.pth 2023-09-04 20:01:33,295 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_88800.pth 2023-09-04 20:01:35,105 44k INFO ====> Epoch: 4343, cost 21.53 s 2023-09-04 20:01:49,721 44k INFO ====> Epoch: 4344, cost 14.62 s 2023-09-04 20:02:04,273 44k INFO ====> Epoch: 4345, cost 14.55 s 2023-09-04 20:02:18,675 44k INFO ====> Epoch: 4346, cost 14.40 s 2023-09-04 20:02:32,837 44k INFO ====> Epoch: 4347, cost 14.16 s 2023-09-04 20:02:47,307 44k INFO ====> Epoch: 4348, cost 14.47 s 2023-09-04 20:03:01,803 44k INFO ====> Epoch: 4349, cost 14.50 s 2023-09-04 20:03:16,221 44k INFO ====> Epoch: 4350, cost 14.42 s 2023-09-04 20:03:30,615 44k INFO ====> Epoch: 4351, cost 14.39 s 2023-09-04 20:03:45,023 44k INFO ====> Epoch: 4352, cost 14.41 s 2023-09-04 20:03:50,658 44k INFO Train Epoch: 4353 [33%] 2023-09-04 20:03:50,658 44k INFO Losses: [2.112226724624634, 2.7113289833068848, 9.175633430480957, 14.728127479553223, 0.607282280921936], step: 91400, lr: 5.8040217948637856e-05, reference_loss: 29.3346004486084 2023-09-04 20:03:59,836 44k INFO ====> Epoch: 4353, cost 14.81 s 2023-09-04 20:04:14,231 44k INFO ====> Epoch: 4354, cost 14.39 s 2023-09-04 20:04:28,697 44k INFO ====> Epoch: 4355, cost 14.47 s 2023-09-04 20:04:42,991 44k INFO ====> Epoch: 4356, cost 14.29 s 2023-09-04 20:04:57,639 44k INFO ====> Epoch: 4357, cost 14.65 s 2023-09-04 20:05:12,270 44k INFO ====> Epoch: 4358, cost 14.63 s 2023-09-04 20:05:26,641 44k INFO ====> Epoch: 4359, cost 14.37 s 2023-09-04 20:05:40,864 44k INFO ====> Epoch: 4360, cost 14.22 s 2023-09-04 20:05:55,665 44k INFO ====> Epoch: 4361, cost 14.80 s 2023-09-04 20:06:09,359 44k INFO Train Epoch: 4362 [86%] 2023-09-04 20:06:09,359 44k INFO Losses: [2.3109476566314697, 3.121526002883911, 8.799739837646484, 15.408644676208496, 0.5557891130447388], step: 91600, lr: 5.797495534154778e-05, reference_loss: 30.19664764404297 2023-09-04 20:06:10,788 44k INFO ====> Epoch: 4362, cost 15.12 s 2023-09-04 20:06:25,279 44k INFO ====> Epoch: 4363, cost 14.49 s 2023-09-04 20:06:39,700 44k INFO ====> Epoch: 4364, cost 14.42 s 2023-09-04 20:06:54,573 44k INFO ====> Epoch: 4365, cost 14.87 s 2023-09-04 20:07:09,174 44k INFO ====> Epoch: 4366, cost 14.60 s 2023-09-04 20:07:23,564 44k INFO ====> Epoch: 4367, cost 14.39 s 2023-09-04 20:07:37,780 44k INFO ====> Epoch: 4368, cost 14.22 s 2023-09-04 20:07:52,442 44k INFO ====> Epoch: 4369, cost 14.66 s 2023-09-04 20:08:07,041 44k INFO ====> Epoch: 4370, cost 14.60 s 2023-09-04 20:08:21,595 44k INFO ====> Epoch: 4371, cost 14.55 s 2023-09-04 20:08:27,913 44k INFO Train Epoch: 4372 [38%] 2023-09-04 20:08:27,913 44k INFO Losses: [2.293982982635498, 2.5588483810424805, 7.961843967437744, 15.829176902770996, 0.43554458022117615], step: 91800, lr: 5.790252739742638e-05, reference_loss: 29.079397201538086 2023-09-04 20:08:36,273 44k INFO ====> Epoch: 4372, cost 14.68 s 2023-09-04 20:08:50,808 44k INFO ====> Epoch: 4373, cost 14.53 s 2023-09-04 20:09:05,302 44k INFO ====> Epoch: 4374, cost 14.49 s 2023-09-04 20:09:19,646 44k INFO ====> Epoch: 4375, cost 14.34 s 2023-09-04 20:09:34,047 44k INFO ====> Epoch: 4376, cost 14.40 s 2023-09-04 20:09:48,493 44k INFO ====> Epoch: 4377, cost 14.44 s 2023-09-04 20:10:03,212 44k INFO ====> Epoch: 4378, cost 14.72 s 2023-09-04 20:10:17,634 44k INFO ====> Epoch: 4379, cost 14.42 s 2023-09-04 20:10:31,833 44k INFO ====> Epoch: 4380, cost 14.20 s 2023-09-04 20:10:45,873 44k INFO Train Epoch: 4381 [90%] 2023-09-04 20:10:45,873 44k INFO Losses: [2.374143362045288, 2.318453073501587, 8.219381332397461, 16.131637573242188, 0.6958584189414978], step: 92000, lr: 5.783741961477806e-05, reference_loss: 29.739473342895508 2023-09-04 20:10:51,369 44k INFO Saving model and optimizer state at iteration 4381 to ./logs\44k\G_92000.pth 2023-09-04 20:10:52,063 44k INFO Saving model and optimizer state at iteration 4381 to ./logs\44k\D_92000.pth 2023-09-04 20:10:53,154 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_89600.pth 2023-09-04 20:10:53,188 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_89600.pth 2023-09-04 20:10:53,650 44k INFO ====> Epoch: 4381, cost 21.82 s 2023-09-04 20:11:08,316 44k INFO ====> Epoch: 4382, cost 14.67 s 2023-09-04 20:11:22,588 44k INFO ====> Epoch: 4383, cost 14.27 s 2023-09-04 20:11:36,823 44k INFO ====> Epoch: 4384, cost 14.23 s 2023-09-04 20:11:51,745 44k INFO ====> Epoch: 4385, cost 14.92 s 2023-09-04 20:12:06,638 44k INFO ====> Epoch: 4386, cost 14.89 s 2023-09-04 20:12:21,392 44k INFO ====> Epoch: 4387, cost 14.75 s 2023-09-04 20:12:36,062 44k INFO ====> Epoch: 4388, cost 14.67 s 2023-09-04 20:12:50,673 44k INFO ====> Epoch: 4389, cost 14.61 s 2023-09-04 20:13:05,424 44k INFO ====> Epoch: 4390, cost 14.75 s 2023-09-04 20:13:12,492 44k INFO Train Epoch: 4391 [43%] 2023-09-04 20:13:12,492 44k INFO Losses: [2.0895447731018066, 3.0511631965637207, 8.388252258300781, 14.841727256774902, 0.5805570483207703], step: 92200, lr: 5.776516349364254e-05, reference_loss: 28.951244354248047 2023-09-04 20:13:20,105 44k INFO ====> Epoch: 4391, cost 14.68 s 2023-09-04 20:13:34,429 44k INFO ====> Epoch: 4392, cost 14.32 s 2023-09-04 20:13:48,915 44k INFO ====> Epoch: 4393, cost 14.49 s 2023-09-04 20:14:03,582 44k INFO ====> Epoch: 4394, cost 14.67 s 2023-09-04 20:14:18,175 44k INFO ====> Epoch: 4395, cost 14.59 s 2023-09-04 20:14:32,525 44k INFO ====> Epoch: 4396, cost 14.35 s 2023-09-04 20:14:47,039 44k INFO ====> Epoch: 4397, cost 14.51 s 2023-09-04 20:15:01,435 44k INFO ====> Epoch: 4398, cost 14.40 s 2023-09-04 20:15:16,006 44k INFO ====> Epoch: 4399, cost 14.57 s 2023-09-04 20:15:30,370 44k INFO Train Epoch: 4400 [95%] 2023-09-04 20:15:30,370 44k INFO Losses: [2.2183618545532227, 2.6852872371673584, 8.760815620422363, 16.240812301635742, 0.20145061612129211], step: 92400, lr: 5.770021016814131e-05, reference_loss: 30.106727600097656 2023-09-04 20:15:30,631 44k INFO ====> Epoch: 4400, cost 14.63 s 2023-09-04 20:15:45,095 44k INFO ====> Epoch: 4401, cost 14.46 s 2023-09-04 20:15:59,572 44k INFO ====> Epoch: 4402, cost 14.48 s 2023-09-04 20:16:14,105 44k INFO ====> Epoch: 4403, cost 14.53 s 2023-09-04 20:16:28,476 44k INFO ====> Epoch: 4404, cost 14.37 s 2023-09-04 20:16:42,777 44k INFO ====> Epoch: 4405, cost 14.30 s 2023-09-04 20:16:57,459 44k INFO ====> Epoch: 4406, cost 14.68 s 2023-09-04 20:17:12,146 44k INFO ====> Epoch: 4407, cost 14.69 s 2023-09-04 20:17:26,450 44k INFO ====> Epoch: 4408, cost 14.30 s 2023-09-04 20:17:40,858 44k INFO ====> Epoch: 4409, cost 14.41 s 2023-09-04 20:17:48,817 44k INFO Train Epoch: 4410 [48%] 2023-09-04 20:17:48,817 44k INFO Losses: [2.2975847721099854, 2.632958173751831, 8.859048843383789, 15.900203704833984, 0.5752333998680115], step: 92600, lr: 5.762812546237085e-05, reference_loss: 30.265029907226562 2023-09-04 20:17:55,982 44k INFO ====> Epoch: 4410, cost 15.12 s 2023-09-04 20:18:10,737 44k INFO ====> Epoch: 4411, cost 14.76 s 2023-09-04 20:18:25,295 44k INFO ====> Epoch: 4412, cost 14.56 s 2023-09-04 20:18:39,806 44k INFO ====> Epoch: 4413, cost 14.51 s 2023-09-04 20:18:54,583 44k INFO ====> Epoch: 4414, cost 14.78 s 2023-09-04 20:19:09,369 44k INFO ====> Epoch: 4415, cost 14.79 s 2023-09-04 20:19:23,704 44k INFO ====> Epoch: 4416, cost 14.34 s 2023-09-04 20:19:37,965 44k INFO ====> Epoch: 4417, cost 14.26 s 2023-09-04 20:19:52,764 44k INFO ====> Epoch: 4418, cost 14.80 s 2023-09-04 20:20:07,490 44k INFO ====> Epoch: 4419, cost 14.73 s 2023-09-04 20:20:08,210 44k INFO Train Epoch: 4420 [0%] 2023-09-04 20:20:08,210 44k INFO Losses: [2.4592056274414062, 2.6631815433502197, 7.082533359527588, 15.537355422973633, 0.5606684684753418], step: 92800, lr: 5.755613081181495e-05, reference_loss: 28.30294418334961 2023-09-04 20:20:13,691 44k INFO Saving model and optimizer state at iteration 4420 to ./logs\44k\G_92800.pth 2023-09-04 20:20:14,328 44k INFO Saving model and optimizer state at iteration 4420 to ./logs\44k\D_92800.pth 2023-09-04 20:20:15,454 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_90400.pth 2023-09-04 20:20:15,492 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_90400.pth 2023-09-04 20:20:29,398 44k INFO ====> Epoch: 4420, cost 21.91 s 2023-09-04 20:20:43,899 44k INFO ====> Epoch: 4421, cost 14.50 s 2023-09-04 20:20:58,571 44k INFO ====> Epoch: 4422, cost 14.67 s 2023-09-04 20:21:13,210 44k INFO ====> Epoch: 4423, cost 14.64 s 2023-09-04 20:21:27,530 44k INFO ====> Epoch: 4424, cost 14.32 s 2023-09-04 20:21:41,814 44k INFO ====> Epoch: 4425, cost 14.28 s 2023-09-04 20:21:56,329 44k INFO ====> Epoch: 4426, cost 14.52 s 2023-09-04 20:22:10,951 44k INFO ====> Epoch: 4427, cost 14.62 s 2023-09-04 20:22:25,347 44k INFO ====> Epoch: 4428, cost 14.40 s 2023-09-04 20:22:33,686 44k INFO Train Epoch: 4429 [52%] 2023-09-04 20:22:33,687 44k INFO Losses: [2.471524477005005, 2.2935824394226074, 8.488972663879395, 16.05640411376953, 0.6488646268844604], step: 93000, lr: 5.7491412530534176e-05, reference_loss: 29.959348678588867 2023-09-04 20:22:39,920 44k INFO ====> Epoch: 4429, cost 14.57 s 2023-09-04 20:22:54,750 44k INFO ====> Epoch: 4430, cost 14.83 s 2023-09-04 20:23:09,410 44k INFO ====> Epoch: 4431, cost 14.66 s 2023-09-04 20:23:23,713 44k INFO ====> Epoch: 4432, cost 14.30 s 2023-09-04 20:23:38,051 44k INFO ====> Epoch: 4433, cost 14.34 s 2023-09-04 20:23:52,821 44k INFO ====> Epoch: 4434, cost 14.77 s 2023-09-04 20:24:07,803 44k INFO ====> Epoch: 4435, cost 14.98 s 2023-09-04 20:24:22,438 44k INFO ====> Epoch: 4436, cost 14.64 s 2023-09-04 20:24:37,075 44k INFO ====> Epoch: 4437, cost 14.64 s 2023-09-04 20:24:51,731 44k INFO ====> Epoch: 4438, cost 14.66 s 2023-09-04 20:24:53,171 44k INFO Train Epoch: 4439 [5%] 2023-09-04 20:24:53,171 44k INFO Losses: [2.2466490268707275, 2.668055295944214, 8.930754661560059, 15.740995407104492, 0.511210560798645], step: 93200, lr: 5.741958867504882e-05, reference_loss: 30.097665786743164 2023-09-04 20:25:06,641 44k INFO ====> Epoch: 4439, cost 14.91 s 2023-09-04 20:25:20,996 44k INFO ====> Epoch: 4440, cost 14.36 s 2023-09-04 20:25:35,203 44k INFO ====> Epoch: 4441, cost 14.21 s 2023-09-04 20:25:49,718 44k INFO ====> Epoch: 4442, cost 14.51 s 2023-09-04 20:26:04,433 44k INFO ====> Epoch: 4443, cost 14.72 s 2023-09-04 20:26:18,861 44k INFO ====> Epoch: 4444, cost 14.43 s 2023-09-04 20:26:33,213 44k INFO ====> Epoch: 4445, cost 14.35 s 2023-09-04 20:26:47,843 44k INFO ====> Epoch: 4446, cost 14.63 s 2023-09-04 20:27:02,425 44k INFO ====> Epoch: 4447, cost 14.58 s 2023-09-04 20:27:11,653 44k INFO Train Epoch: 4448 [57%] 2023-09-04 20:27:11,653 44k INFO Losses: [2.2251296043395996, 2.666360855102539, 8.041842460632324, 15.637859344482422, 0.503440797328949], step: 93400, lr: 5.735502392688936e-05, reference_loss: 29.07463264465332 2023-09-04 20:27:17,164 44k INFO ====> Epoch: 4448, cost 14.74 s 2023-09-04 20:27:31,297 44k INFO ====> Epoch: 4449, cost 14.13 s 2023-09-04 20:27:45,793 44k INFO ====> Epoch: 4450, cost 14.50 s 2023-09-04 20:28:00,254 44k INFO ====> Epoch: 4451, cost 14.46 s 2023-09-04 20:28:14,808 44k INFO ====> Epoch: 4452, cost 14.55 s 2023-09-04 20:28:29,160 44k INFO ====> Epoch: 4453, cost 14.35 s 2023-09-04 20:28:43,677 44k INFO ====> Epoch: 4454, cost 14.52 s 2023-09-04 20:28:58,269 44k INFO ====> Epoch: 4455, cost 14.59 s 2023-09-04 20:29:12,801 44k INFO ====> Epoch: 4456, cost 14.53 s 2023-09-04 20:29:26,953 44k INFO ====> Epoch: 4457, cost 14.15 s 2023-09-04 20:29:29,012 44k INFO Train Epoch: 4458 [10%] 2023-09-04 20:29:29,013 44k INFO Losses: [2.2992103099823, 2.6080777645111084, 8.206461906433105, 14.733475685119629, 0.6808427572250366], step: 93600, lr: 5.728337046129228e-05, reference_loss: 28.52806854248047 2023-09-04 20:29:34,419 44k INFO Saving model and optimizer state at iteration 4458 to ./logs\44k\G_93600.pth 2023-09-04 20:29:35,096 44k INFO Saving model and optimizer state at iteration 4458 to ./logs\44k\D_93600.pth 2023-09-04 20:29:36,177 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_91200.pth 2023-09-04 20:29:36,212 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_91200.pth 2023-09-04 20:29:48,797 44k INFO ====> Epoch: 4458, cost 21.84 s 2023-09-04 20:30:03,597 44k INFO ====> Epoch: 4459, cost 14.80 s 2023-09-04 20:30:18,409 44k INFO ====> Epoch: 4460, cost 14.81 s 2023-09-04 20:30:32,959 44k INFO ====> Epoch: 4461, cost 14.55 s 2023-09-04 20:30:47,528 44k INFO ====> Epoch: 4462, cost 14.57 s 2023-09-04 20:31:02,323 44k INFO ====> Epoch: 4463, cost 14.79 s 2023-09-04 20:31:16,976 44k INFO ====> Epoch: 4464, cost 14.65 s 2023-09-04 20:31:31,151 44k INFO ====> Epoch: 4465, cost 14.17 s 2023-09-04 20:31:45,638 44k INFO ====> Epoch: 4466, cost 14.49 s 2023-09-04 20:31:55,686 44k INFO Train Epoch: 4467 [62%] 2023-09-04 20:31:55,687 44k INFO Losses: [2.3891818523406982, 2.6056740283966064, 7.945308208465576, 16.681882858276367, 0.44693711400032043], step: 93800, lr: 5.7218958882022896e-05, reference_loss: 30.068984985351562 2023-09-04 20:32:00,690 44k INFO ====> Epoch: 4467, cost 15.05 s 2023-09-04 20:32:15,345 44k INFO ====> Epoch: 4468, cost 14.65 s 2023-09-04 20:32:29,704 44k INFO ====> Epoch: 4469, cost 14.36 s 2023-09-04 20:32:44,277 44k INFO ====> Epoch: 4470, cost 14.57 s 2023-09-04 20:32:58,759 44k INFO ====> Epoch: 4471, cost 14.48 s 2023-09-04 20:33:13,287 44k INFO ====> Epoch: 4472, cost 14.53 s 2023-09-04 20:33:27,723 44k INFO ====> Epoch: 4473, cost 14.44 s 2023-09-04 20:33:42,361 44k INFO ====> Epoch: 4474, cost 14.64 s 2023-09-04 20:33:56,854 44k INFO ====> Epoch: 4475, cost 14.49 s 2023-09-04 20:34:11,532 44k INFO ====> Epoch: 4476, cost 14.68 s 2023-09-04 20:34:14,299 44k INFO Train Epoch: 4477 [14%] 2023-09-04 20:34:14,300 44k INFO Losses: [2.4713804721832275, 2.379969835281372, 8.178878784179688, 16.318775177001953, 0.5429845452308655], step: 94000, lr: 5.714747540209304e-05, reference_loss: 29.89198875427246 2023-09-04 20:34:26,249 44k INFO ====> Epoch: 4477, cost 14.72 s 2023-09-04 20:34:40,809 44k INFO ====> Epoch: 4478, cost 14.56 s 2023-09-04 20:34:55,555 44k INFO ====> Epoch: 4479, cost 14.75 s 2023-09-04 20:35:10,057 44k INFO ====> Epoch: 4480, cost 14.50 s 2023-09-04 20:35:24,267 44k INFO ====> Epoch: 4481, cost 14.21 s 2023-09-04 20:35:38,629 44k INFO ====> Epoch: 4482, cost 14.36 s 2023-09-04 20:35:53,415 44k INFO ====> Epoch: 4483, cost 14.79 s 2023-09-04 20:36:08,228 44k INFO ====> Epoch: 4484, cost 14.81 s 2023-09-04 20:36:22,767 44k INFO ====> Epoch: 4485, cost 14.54 s 2023-09-04 20:36:33,319 44k INFO Train Epoch: 4486 [67%] 2023-09-04 20:36:33,319 44k INFO Losses: [2.271519422531128, 2.4338126182556152, 7.002492904663086, 14.387395858764648, 0.5058192014694214], step: 94200, lr: 5.708321662834658e-05, reference_loss: 26.60103988647461 2023-09-04 20:36:37,445 44k INFO ====> Epoch: 4486, cost 14.68 s 2023-09-04 20:36:52,076 44k INFO ====> Epoch: 4487, cost 14.63 s 2023-09-04 20:37:06,812 44k INFO ====> Epoch: 4488, cost 14.74 s 2023-09-04 20:37:21,264 44k INFO ====> Epoch: 4489, cost 14.45 s 2023-09-04 20:37:35,618 44k INFO ====> Epoch: 4490, cost 14.35 s 2023-09-04 20:37:50,161 44k INFO ====> Epoch: 4491, cost 14.54 s 2023-09-04 20:38:04,873 44k INFO ====> Epoch: 4492, cost 14.71 s 2023-09-04 20:38:19,366 44k INFO ====> Epoch: 4493, cost 14.49 s 2023-09-04 20:38:34,032 44k INFO ====> Epoch: 4494, cost 14.67 s 2023-09-04 20:38:48,664 44k INFO ====> Epoch: 4495, cost 14.63 s 2023-09-04 20:38:52,222 44k INFO Train Epoch: 4496 [19%] 2023-09-04 20:38:52,222 44k INFO Losses: [2.27001690864563, 2.696481943130493, 8.573712348937988, 15.659981727600098, 0.6597650051116943], step: 94400, lr: 5.701190273082185e-05, reference_loss: 29.859956741333008 2023-09-04 20:38:58,317 44k INFO Saving model and optimizer state at iteration 4496 to ./logs\44k\G_94400.pth 2023-09-04 20:38:58,927 44k INFO Saving model and optimizer state at iteration 4496 to ./logs\44k\D_94400.pth 2023-09-04 20:39:00,083 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_92000.pth 2023-09-04 20:39:00,114 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_92000.pth 2023-09-04 20:39:11,075 44k INFO ====> Epoch: 4496, cost 22.41 s 2023-09-04 20:39:25,570 44k INFO ====> Epoch: 4497, cost 14.50 s 2023-09-04 20:39:40,224 44k INFO ====> Epoch: 4498, cost 14.65 s 2023-09-04 20:39:54,700 44k INFO ====> Epoch: 4499, cost 14.48 s 2023-09-04 20:40:09,349 44k INFO ====> Epoch: 4500, cost 14.65 s 2023-09-04 20:40:23,832 44k INFO ====> Epoch: 4501, cost 14.48 s 2023-09-04 20:40:38,121 44k INFO ====> Epoch: 4502, cost 14.29 s 2023-09-04 20:40:52,744 44k INFO ====> Epoch: 4503, cost 14.62 s 2023-09-04 20:41:07,213 44k INFO ====> Epoch: 4504, cost 14.47 s 2023-09-04 20:41:18,237 44k INFO Train Epoch: 4505 [71%] 2023-09-04 20:41:18,237 44k INFO Losses: [2.1109251976013184, 3.0725831985473633, 7.695714950561523, 13.935609817504883, 0.7215238809585571], step: 94600, lr: 5.694779640009318e-05, reference_loss: 27.53635597229004 2023-09-04 20:41:21,665 44k INFO ====> Epoch: 4505, cost 14.45 s 2023-09-04 20:41:35,952 44k INFO ====> Epoch: 4506, cost 14.29 s 2023-09-04 20:41:50,709 44k INFO ====> Epoch: 4507, cost 14.76 s 2023-09-04 20:42:05,421 44k INFO ====> Epoch: 4508, cost 14.71 s 2023-09-04 20:42:20,060 44k INFO ====> Epoch: 4509, cost 14.64 s 2023-09-04 20:42:34,477 44k INFO ====> Epoch: 4510, cost 14.42 s 2023-09-04 20:42:49,211 44k INFO ====> Epoch: 4511, cost 14.73 s 2023-09-04 20:43:03,846 44k INFO ====> Epoch: 4512, cost 14.64 s 2023-09-04 20:43:18,251 44k INFO ====> Epoch: 4513, cost 14.40 s 2023-09-04 20:43:32,364 44k INFO ====> Epoch: 4514, cost 14.11 s 2023-09-04 20:43:36,484 44k INFO Train Epoch: 4515 [24%] 2023-09-04 20:43:36,485 44k INFO Losses: [2.571336507797241, 2.3159899711608887, 7.168062210083008, 14.879908561706543, 0.5034640431404114], step: 94800, lr: 5.6876651682668165e-05, reference_loss: 27.43876075744629 2023-09-04 20:43:47,098 44k INFO ====> Epoch: 4515, cost 14.73 s 2023-09-04 20:44:01,744 44k INFO ====> Epoch: 4516, cost 14.65 s 2023-09-04 20:44:16,237 44k INFO ====> Epoch: 4517, cost 14.49 s 2023-09-04 20:44:30,735 44k INFO ====> Epoch: 4518, cost 14.50 s 2023-09-04 20:44:45,288 44k INFO ====> Epoch: 4519, cost 14.55 s 2023-09-04 20:44:59,942 44k INFO ====> Epoch: 4520, cost 14.65 s 2023-09-04 20:45:14,378 44k INFO ====> Epoch: 4521, cost 14.44 s 2023-09-04 20:45:28,568 44k INFO ====> Epoch: 4522, cost 14.19 s 2023-09-04 20:45:43,030 44k INFO ====> Epoch: 4523, cost 14.46 s 2023-09-04 20:45:55,005 44k INFO Train Epoch: 4524 [76%] 2023-09-04 20:45:55,005 44k INFO Losses: [1.9849718809127808, 2.9449760913848877, 8.451075553894043, 13.791902542114258, 0.47074759006500244], step: 95000, lr: 5.6812697433312136e-05, reference_loss: 27.643672943115234 2023-09-04 20:45:57,733 44k INFO ====> Epoch: 4524, cost 14.70 s 2023-09-04 20:46:12,389 44k INFO ====> Epoch: 4525, cost 14.66 s 2023-09-04 20:46:26,806 44k INFO ====> Epoch: 4526, cost 14.42 s 2023-09-04 20:46:41,361 44k INFO ====> Epoch: 4527, cost 14.55 s 2023-09-04 20:46:56,047 44k INFO ====> Epoch: 4528, cost 14.69 s 2023-09-04 20:47:10,885 44k INFO ====> Epoch: 4529, cost 14.84 s 2023-09-04 20:47:25,258 44k INFO ====> Epoch: 4530, cost 14.37 s 2023-09-04 20:47:39,440 44k INFO ====> Epoch: 4531, cost 14.18 s 2023-09-04 20:47:54,154 44k INFO ====> Epoch: 4532, cost 14.71 s 2023-09-04 20:48:09,070 44k INFO ====> Epoch: 4533, cost 14.92 s 2023-09-04 20:48:14,015 44k INFO Train Epoch: 4534 [29%] 2023-09-04 20:48:14,015 44k INFO Losses: [2.2517447471618652, 2.6701064109802246, 11.661158561706543, 17.46363639831543, 0.7327433228492737], step: 95200, lr: 5.6741721494635776e-05, reference_loss: 34.779388427734375 2023-09-04 20:48:19,425 44k INFO Saving model and optimizer state at iteration 4534 to ./logs\44k\G_95200.pth 2023-09-04 20:48:20,060 44k INFO Saving model and optimizer state at iteration 4534 to ./logs\44k\D_95200.pth 2023-09-04 20:48:21,193 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_92800.pth 2023-09-04 20:48:21,224 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_92800.pth 2023-09-04 20:48:30,814 44k INFO ====> Epoch: 4534, cost 21.74 s 2023-09-04 20:48:45,403 44k INFO ====> Epoch: 4535, cost 14.59 s 2023-09-04 20:48:59,955 44k INFO ====> Epoch: 4536, cost 14.55 s 2023-09-04 20:49:14,548 44k INFO ====> Epoch: 4537, cost 14.59 s 2023-09-04 20:49:28,705 44k INFO ====> Epoch: 4538, cost 14.16 s 2023-09-04 20:49:43,057 44k INFO ====> Epoch: 4539, cost 14.35 s 2023-09-04 20:49:57,911 44k INFO ====> Epoch: 4540, cost 14.85 s 2023-09-04 20:50:12,550 44k INFO ====> Epoch: 4541, cost 14.64 s 2023-09-04 20:50:26,868 44k INFO ====> Epoch: 4542, cost 14.32 s 2023-09-04 20:50:39,592 44k INFO Train Epoch: 4543 [81%] 2023-09-04 20:50:39,593 44k INFO Losses: [2.089016914367676, 2.921097755432129, 9.969307899475098, 16.57794189453125, 0.6456644535064697], step: 95400, lr: 5.6677918965865176e-05, reference_loss: 32.20302963256836 2023-09-04 20:50:41,721 44k INFO ====> Epoch: 4543, cost 14.85 s 2023-09-04 20:50:56,351 44k INFO ====> Epoch: 4544, cost 14.63 s 2023-09-04 20:51:10,872 44k INFO ====> Epoch: 4545, cost 14.52 s 2023-09-04 20:51:25,205 44k INFO ====> Epoch: 4546, cost 14.33 s 2023-09-04 20:51:39,594 44k INFO ====> Epoch: 4547, cost 14.39 s 2023-09-04 20:51:54,280 44k INFO ====> Epoch: 4548, cost 14.69 s 2023-09-04 20:52:08,926 44k INFO ====> Epoch: 4549, cost 14.65 s 2023-09-04 20:52:23,199 44k INFO ====> Epoch: 4550, cost 14.27 s 2023-09-04 20:52:37,414 44k INFO ====> Epoch: 4551, cost 14.22 s 2023-09-04 20:52:51,995 44k INFO ====> Epoch: 4552, cost 14.58 s 2023-09-04 20:52:57,754 44k INFO Train Epoch: 4553 [33%] 2023-09-04 20:52:57,754 44k INFO Losses: [2.414304256439209, 2.3840909004211426, 8.773916244506836, 17.294387817382812, 0.5945044159889221], step: 95600, lr: 5.660711140553862e-05, reference_loss: 31.461204528808594 2023-09-04 20:53:06,864 44k INFO ====> Epoch: 4553, cost 14.87 s 2023-09-04 20:53:21,288 44k INFO ====> Epoch: 4554, cost 14.42 s 2023-09-04 20:53:35,636 44k INFO ====> Epoch: 4555, cost 14.35 s 2023-09-04 20:53:50,216 44k INFO ====> Epoch: 4556, cost 14.58 s 2023-09-04 20:54:04,916 44k INFO ====> Epoch: 4557, cost 14.70 s 2023-09-04 20:54:19,494 44k INFO ====> Epoch: 4558, cost 14.58 s 2023-09-04 20:54:33,948 44k INFO ====> Epoch: 4559, cost 14.45 s 2023-09-04 20:54:48,628 44k INFO ====> Epoch: 4560, cost 14.68 s 2023-09-04 20:55:03,366 44k INFO ====> Epoch: 4561, cost 14.74 s 2023-09-04 20:55:16,914 44k INFO Train Epoch: 4562 [86%] 2023-09-04 20:55:16,914 44k INFO Losses: [2.10378098487854, 2.940558910369873, 9.301325798034668, 14.557151794433594, 0.6406141519546509], step: 95800, lr: 5.654346023742217e-05, reference_loss: 29.54343032836914 2023-09-04 20:55:18,292 44k INFO ====> Epoch: 4562, cost 14.93 s 2023-09-04 20:55:32,445 44k INFO ====> Epoch: 4563, cost 14.15 s 2023-09-04 20:55:46,828 44k INFO ====> Epoch: 4564, cost 14.38 s 2023-09-04 20:56:01,403 44k INFO ====> Epoch: 4565, cost 14.58 s 2023-09-04 20:56:15,928 44k INFO ====> Epoch: 4566, cost 14.52 s 2023-09-04 20:56:30,418 44k INFO ====> Epoch: 4567, cost 14.49 s 2023-09-04 20:56:45,081 44k INFO ====> Epoch: 4568, cost 14.66 s 2023-09-04 20:56:59,832 44k INFO ====> Epoch: 4569, cost 14.75 s 2023-09-04 20:57:14,262 44k INFO ====> Epoch: 4570, cost 14.43 s 2023-09-04 20:57:28,662 44k INFO ====> Epoch: 4571, cost 14.40 s 2023-09-04 20:57:34,890 44k INFO Train Epoch: 4572 [38%] 2023-09-04 20:57:34,890 44k INFO Losses: [2.6402933597564697, 2.26881742477417, 6.362489700317383, 14.27411937713623, 0.5906813144683838], step: 96000, lr: 5.647282065599637e-05, reference_loss: 26.13640022277832 2023-09-04 20:57:40,260 44k INFO Saving model and optimizer state at iteration 4572 to ./logs\44k\G_96000.pth 2023-09-04 20:57:40,883 44k INFO Saving model and optimizer state at iteration 4572 to ./logs\44k\D_96000.pth 2023-09-04 20:57:42,115 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_93600.pth 2023-09-04 20:57:42,148 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_93600.pth 2023-09-04 20:57:50,332 44k INFO ====> Epoch: 4572, cost 21.67 s 2023-09-04 20:58:04,917 44k INFO ====> Epoch: 4573, cost 14.58 s 2023-09-04 20:58:19,366 44k INFO ====> Epoch: 4574, cost 14.45 s 2023-09-04 20:58:33,779 44k INFO ====> Epoch: 4575, cost 14.41 s 2023-09-04 20:58:48,247 44k INFO ====> Epoch: 4576, cost 14.47 s 2023-09-04 20:59:02,988 44k INFO ====> Epoch: 4577, cost 14.74 s 2023-09-04 20:59:17,401 44k INFO ====> Epoch: 4578, cost 14.41 s 2023-09-04 20:59:31,683 44k INFO ====> Epoch: 4579, cost 14.28 s 2023-09-04 20:59:46,158 44k INFO ====> Epoch: 4580, cost 14.47 s 2023-09-04 21:00:00,722 44k INFO Train Epoch: 4581 [90%] 2023-09-04 21:00:00,722 44k INFO Losses: [2.2317051887512207, 2.63071608543396, 8.382213592529297, 16.23371124267578, 0.782650887966156], step: 96200, lr: 5.640932048945664e-05, reference_loss: 30.260995864868164 2023-09-04 21:00:01,412 44k INFO ====> Epoch: 4581, cost 15.25 s 2023-09-04 21:00:15,889 44k INFO ====> Epoch: 4582, cost 14.48 s 2023-09-04 21:00:30,146 44k INFO ====> Epoch: 4583, cost 14.26 s 2023-09-04 21:00:44,485 44k INFO ====> Epoch: 4584, cost 14.34 s 2023-09-04 21:00:59,355 44k INFO ====> Epoch: 4585, cost 14.87 s 2023-09-04 21:01:13,950 44k INFO ====> Epoch: 4586, cost 14.60 s 2023-09-04 21:01:28,332 44k INFO ====> Epoch: 4587, cost 14.38 s 2023-09-04 21:01:42,791 44k INFO ====> Epoch: 4588, cost 14.46 s 2023-09-04 21:01:57,380 44k INFO ====> Epoch: 4589, cost 14.59 s 2023-09-04 21:02:11,964 44k INFO ====> Epoch: 4590, cost 14.58 s 2023-09-04 21:02:19,027 44k INFO Train Epoch: 4591 [43%] 2023-09-04 21:02:19,028 44k INFO Losses: [2.4831275939941406, 2.1308095455169678, 6.758296966552734, 14.56886100769043, 0.5319671630859375], step: 96400, lr: 5.633884848843023e-05, reference_loss: 26.47306251525879 2023-09-04 21:02:26,814 44k INFO ====> Epoch: 4591, cost 14.85 s 2023-09-04 21:02:41,240 44k INFO ====> Epoch: 4592, cost 14.43 s 2023-09-04 21:02:55,760 44k INFO ====> Epoch: 4593, cost 14.52 s 2023-09-04 21:03:10,286 44k INFO ====> Epoch: 4594, cost 14.53 s 2023-09-04 21:03:24,528 44k INFO ====> Epoch: 4595, cost 14.24 s 2023-09-04 21:03:38,957 44k INFO ====> Epoch: 4596, cost 14.43 s 2023-09-04 21:03:53,315 44k INFO ====> Epoch: 4597, cost 14.36 s 2023-09-04 21:04:07,951 44k INFO ====> Epoch: 4598, cost 14.64 s 2023-09-04 21:04:22,279 44k INFO ====> Epoch: 4599, cost 14.33 s 2023-09-04 21:04:36,673 44k INFO Train Epoch: 4600 [95%] 2023-09-04 21:04:36,674 44k INFO Losses: [2.0118985176086426, 3.0805275440216064, 11.83411979675293, 15.081023216247559, 0.5211343765258789], step: 96600, lr: 5.6275498965241634e-05, reference_loss: 32.52870178222656 2023-09-04 21:04:36,934 44k INFO ====> Epoch: 4600, cost 14.66 s 2023-09-04 21:04:51,550 44k INFO ====> Epoch: 4601, cost 14.62 s 2023-09-04 21:05:06,206 44k INFO ====> Epoch: 4602, cost 14.66 s 2023-09-04 21:05:20,733 44k INFO ====> Epoch: 4603, cost 14.53 s 2023-09-04 21:05:35,013 44k INFO ====> Epoch: 4604, cost 14.28 s 2023-09-04 21:05:49,623 44k INFO ====> Epoch: 4605, cost 14.61 s 2023-09-04 21:06:04,504 44k INFO ====> Epoch: 4606, cost 14.88 s 2023-09-04 21:06:19,163 44k INFO ====> Epoch: 4607, cost 14.66 s 2023-09-04 21:06:33,425 44k INFO ====> Epoch: 4608, cost 14.26 s 2023-09-04 21:06:48,062 44k INFO ====> Epoch: 4609, cost 14.64 s 2023-09-04 21:06:55,970 44k INFO Train Epoch: 4610 [48%] 2023-09-04 21:06:55,970 44k INFO Losses: [2.159882068634033, 2.7162561416625977, 10.492359161376953, 17.013866424560547, 0.6441375613212585], step: 96800, lr: 5.620519414705858e-05, reference_loss: 33.0265007019043 2023-09-04 21:07:01,434 44k INFO Saving model and optimizer state at iteration 4610 to ./logs\44k\G_96800.pth 2023-09-04 21:07:02,050 44k INFO Saving model and optimizer state at iteration 4610 to ./logs\44k\D_96800.pth 2023-09-04 21:07:03,653 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_94400.pth 2023-09-04 21:07:03,686 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_94400.pth 2023-09-04 21:07:10,735 44k INFO ====> Epoch: 4610, cost 22.67 s 2023-09-04 21:07:25,013 44k INFO ====> Epoch: 4611, cost 14.28 s 2023-09-04 21:07:39,163 44k INFO ====> Epoch: 4612, cost 14.15 s 2023-09-04 21:07:53,818 44k INFO ====> Epoch: 4613, cost 14.66 s 2023-09-04 21:08:08,471 44k INFO ====> Epoch: 4614, cost 14.65 s 2023-09-04 21:08:22,886 44k INFO ====> Epoch: 4615, cost 14.42 s 2023-09-04 21:08:37,373 44k INFO ====> Epoch: 4616, cost 14.49 s 2023-09-04 21:08:51,991 44k INFO ====> Epoch: 4617, cost 14.62 s 2023-09-04 21:09:06,424 44k INFO ====> Epoch: 4618, cost 14.43 s 2023-09-04 21:09:20,934 44k INFO ====> Epoch: 4619, cost 14.51 s 2023-09-04 21:09:21,618 44k INFO Train Epoch: 4620 [0%] 2023-09-04 21:09:21,619 44k INFO Losses: [2.035132884979248, 3.1716508865356445, 9.321174621582031, 15.431726455688477, 0.5661345720291138], step: 97000, lr: 5.6134977160481675e-05, reference_loss: 30.525819778442383 2023-09-04 21:09:35,365 44k INFO ====> Epoch: 4620, cost 14.43 s 2023-09-04 21:09:50,061 44k INFO ====> Epoch: 4621, cost 14.70 s 2023-09-04 21:10:04,537 44k INFO ====> Epoch: 4622, cost 14.48 s 2023-09-04 21:10:19,040 44k INFO ====> Epoch: 4623, cost 14.50 s 2023-09-04 21:10:33,362 44k INFO ====> Epoch: 4624, cost 14.32 s 2023-09-04 21:10:47,834 44k INFO ====> Epoch: 4625, cost 14.47 s 2023-09-04 21:11:02,459 44k INFO ====> Epoch: 4626, cost 14.63 s 2023-09-04 21:11:16,920 44k INFO ====> Epoch: 4627, cost 14.46 s 2023-09-04 21:11:31,078 44k INFO ====> Epoch: 4628, cost 14.16 s 2023-09-04 21:11:39,376 44k INFO Train Epoch: 4629 [52%] 2023-09-04 21:11:39,376 44k INFO Losses: [2.472390651702881, 2.5771682262420654, 7.149517059326172, 15.67933464050293, 0.6601218581199646], step: 97200, lr: 5.6071856877892845e-05, reference_loss: 28.538532257080078 2023-09-04 21:11:45,878 44k INFO ====> Epoch: 4629, cost 14.80 s 2023-09-04 21:12:00,745 44k INFO ====> Epoch: 4630, cost 14.87 s 2023-09-04 21:12:15,334 44k INFO ====> Epoch: 4631, cost 14.59 s 2023-09-04 21:12:29,707 44k INFO ====> Epoch: 4632, cost 14.37 s 2023-09-04 21:12:44,180 44k INFO ====> Epoch: 4633, cost 14.47 s 2023-09-04 21:12:58,911 44k INFO ====> Epoch: 4634, cost 14.73 s 2023-09-04 21:13:13,534 44k INFO ====> Epoch: 4635, cost 14.62 s 2023-09-04 21:13:27,918 44k INFO ====> Epoch: 4636, cost 14.38 s 2023-09-04 21:13:42,407 44k INFO ====> Epoch: 4637, cost 14.49 s 2023-09-04 21:13:56,989 44k INFO ====> Epoch: 4638, cost 14.58 s 2023-09-04 21:13:58,452 44k INFO Train Epoch: 4639 [5%] 2023-09-04 21:13:58,452 44k INFO Losses: [2.16044282913208, 2.5871198177337646, 7.7626423835754395, 16.23992919921875, 0.5580320954322815], step: 97400, lr: 5.600180646918086e-05, reference_loss: 29.30816650390625 2023-09-04 21:14:12,069 44k INFO ====> Epoch: 4639, cost 15.08 s 2023-09-04 21:14:26,262 44k INFO ====> Epoch: 4640, cost 14.19 s 2023-09-04 21:14:40,752 44k INFO ====> Epoch: 4641, cost 14.49 s 2023-09-04 21:14:55,330 44k INFO ====> Epoch: 4642, cost 14.58 s 2023-09-04 21:15:10,024 44k INFO ====> Epoch: 4643, cost 14.69 s 2023-09-04 21:15:24,376 44k INFO ====> Epoch: 4644, cost 14.35 s 2023-09-04 21:15:38,666 44k INFO ====> Epoch: 4645, cost 14.29 s 2023-09-04 21:15:53,053 44k INFO ====> Epoch: 4646, cost 14.39 s 2023-09-04 21:16:07,707 44k INFO ====> Epoch: 4647, cost 14.65 s 2023-09-04 21:16:16,888 44k INFO Train Epoch: 4648 [57%] 2023-09-04 21:16:16,889 44k INFO Losses: [2.1592588424682617, 2.8988661766052246, 9.239011764526367, 15.955572128295898, 0.7847681045532227], step: 97600, lr: 5.593883592873308e-05, reference_loss: 31.0374755859375 2023-09-04 21:16:22,272 44k INFO Saving model and optimizer state at iteration 4648 to ./logs\44k\G_97600.pth 2023-09-04 21:16:22,929 44k INFO Saving model and optimizer state at iteration 4648 to ./logs\44k\D_97600.pth 2023-09-04 21:16:24,150 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_95200.pth 2023-09-04 21:16:24,191 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_95200.pth 2023-09-04 21:16:29,523 44k INFO ====> Epoch: 4648, cost 21.82 s 2023-09-04 21:16:43,934 44k INFO ====> Epoch: 4649, cost 14.41 s 2023-09-04 21:16:58,726 44k INFO ====> Epoch: 4650, cost 14.79 s 2023-09-04 21:17:13,416 44k INFO ====> Epoch: 4651, cost 14.69 s 2023-09-04 21:17:27,847 44k INFO ====> Epoch: 4652, cost 14.43 s 2023-09-04 21:17:42,074 44k INFO ====> Epoch: 4653, cost 14.23 s 2023-09-04 21:17:56,800 44k INFO ====> Epoch: 4654, cost 14.73 s 2023-09-04 21:18:11,592 44k INFO ====> Epoch: 4655, cost 14.79 s 2023-09-04 21:18:25,937 44k INFO ====> Epoch: 4656, cost 14.34 s 2023-09-04 21:18:40,372 44k INFO ====> Epoch: 4657, cost 14.43 s 2023-09-04 21:18:42,490 44k INFO Train Epoch: 4658 [10%] 2023-09-04 21:18:42,491 44k INFO Losses: [1.959747076034546, 3.1022188663482666, 11.013752937316895, 15.120884895324707, 0.654259443283081], step: 97800, lr: 5.586895170270835e-05, reference_loss: 31.850860595703125 2023-09-04 21:18:55,372 44k INFO ====> Epoch: 4658, cost 15.00 s 2023-09-04 21:19:10,253 44k INFO ====> Epoch: 4659, cost 14.88 s 2023-09-04 21:19:24,500 44k INFO ====> Epoch: 4660, cost 14.25 s 2023-09-04 21:19:38,979 44k INFO ====> Epoch: 4661, cost 14.48 s 2023-09-04 21:19:53,796 44k INFO ====> Epoch: 4662, cost 14.82 s 2023-09-04 21:20:08,382 44k INFO ====> Epoch: 4663, cost 14.59 s 2023-09-04 21:20:22,981 44k INFO ====> Epoch: 4664, cost 14.60 s 2023-09-04 21:20:37,405 44k INFO ====> Epoch: 4665, cost 14.42 s 2023-09-04 21:20:51,933 44k INFO ====> Epoch: 4666, cost 14.53 s 2023-09-04 21:21:01,847 44k INFO Train Epoch: 4667 [62%] 2023-09-04 21:21:01,848 44k INFO Losses: [2.2672338485717773, 2.5122387409210205, 9.175378799438477, 17.113191604614258, 0.5353674292564392], step: 98000, lr: 5.5806130549163826e-05, reference_loss: 31.603410720825195 2023-09-04 21:21:06,713 44k INFO ====> Epoch: 4667, cost 14.78 s 2023-09-04 21:21:21,101 44k INFO ====> Epoch: 4668, cost 14.39 s 2023-09-04 21:21:35,354 44k INFO ====> Epoch: 4669, cost 14.25 s 2023-09-04 21:21:49,964 44k INFO ====> Epoch: 4670, cost 14.61 s 2023-09-04 21:22:04,464 44k INFO ====> Epoch: 4671, cost 14.50 s 2023-09-04 21:22:18,733 44k INFO ====> Epoch: 4672, cost 14.27 s 2023-09-04 21:22:33,124 44k INFO ====> Epoch: 4673, cost 14.39 s 2023-09-04 21:22:47,652 44k INFO ====> Epoch: 4674, cost 14.53 s 2023-09-04 21:23:02,222 44k INFO ====> Epoch: 4675, cost 14.57 s 2023-09-04 21:23:16,640 44k INFO ====> Epoch: 4676, cost 14.42 s 2023-09-04 21:23:19,474 44k INFO Train Epoch: 4677 [14%] 2023-09-04 21:23:19,474 44k INFO Losses: [2.2690906524658203, 2.7422924041748047, 8.993943214416504, 15.759233474731445, 0.5865442156791687], step: 98200, lr: 5.573641211158618e-05, reference_loss: 30.351104736328125 2023-09-04 21:23:31,228 44k INFO ====> Epoch: 4677, cost 14.59 s 2023-09-04 21:23:45,491 44k INFO ====> Epoch: 4678, cost 14.26 s 2023-09-04 21:24:00,316 44k INFO ====> Epoch: 4679, cost 14.82 s 2023-09-04 21:24:15,194 44k INFO ====> Epoch: 4680, cost 14.88 s 2023-09-04 21:24:29,582 44k INFO ====> Epoch: 4681, cost 14.39 s 2023-09-04 21:24:44,182 44k INFO ====> Epoch: 4682, cost 14.60 s 2023-09-04 21:24:59,057 44k INFO ====> Epoch: 4683, cost 14.87 s 2023-09-04 21:25:13,553 44k INFO ====> Epoch: 4684, cost 14.50 s 2023-09-04 21:25:27,754 44k INFO ====> Epoch: 4685, cost 14.20 s 2023-09-04 21:25:38,138 44k INFO Train Epoch: 4686 [67%] 2023-09-04 21:25:38,138 44k INFO Losses: [2.153355598449707, 2.654317855834961, 8.23167610168457, 16.8619384765625, 0.5944791436195374], step: 98400, lr: 5.5673739990549895e-05, reference_loss: 30.495765686035156 2023-09-04 21:25:43,603 44k INFO Saving model and optimizer state at iteration 4686 to ./logs\44k\G_98400.pth 2023-09-04 21:25:44,275 44k INFO Saving model and optimizer state at iteration 4686 to ./logs\44k\D_98400.pth 2023-09-04 21:25:45,357 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_96000.pth 2023-09-04 21:25:45,392 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_96000.pth 2023-09-04 21:25:49,390 44k INFO ====> Epoch: 4686, cost 21.64 s 2023-09-04 21:26:03,930 44k INFO ====> Epoch: 4687, cost 14.54 s 2023-09-04 21:26:18,387 44k INFO ====> Epoch: 4688, cost 14.46 s 2023-09-04 21:26:32,688 44k INFO ====> Epoch: 4689, cost 14.30 s 2023-09-04 21:26:47,189 44k INFO ====> Epoch: 4690, cost 14.50 s 2023-09-04 21:27:01,649 44k INFO ====> Epoch: 4691, cost 14.46 s 2023-09-04 21:27:16,310 44k INFO ====> Epoch: 4692, cost 14.66 s 2023-09-04 21:27:30,740 44k INFO ====> Epoch: 4693, cost 14.43 s 2023-09-04 21:27:45,157 44k INFO ====> Epoch: 4694, cost 14.42 s 2023-09-04 21:27:59,718 44k INFO ====> Epoch: 4695, cost 14.56 s 2023-09-04 21:28:03,295 44k INFO Train Epoch: 4696 [19%] 2023-09-04 21:28:03,296 44k INFO Losses: [2.3230671882629395, 2.3560800552368164, 8.579367637634277, 16.71514892578125, 0.6840106844902039], step: 98600, lr: 5.5604186948114444e-05, reference_loss: 30.65767478942871 2023-09-04 21:28:14,474 44k INFO ====> Epoch: 4696, cost 14.76 s 2023-09-04 21:28:28,829 44k INFO ====> Epoch: 4697, cost 14.36 s 2023-09-04 21:28:43,139 44k INFO ====> Epoch: 4698, cost 14.31 s 2023-09-04 21:28:57,747 44k INFO ====> Epoch: 4699, cost 14.61 s 2023-09-04 21:29:12,352 44k INFO ====> Epoch: 4700, cost 14.61 s 2023-09-04 21:29:26,792 44k INFO ====> Epoch: 4701, cost 14.44 s 2023-09-04 21:29:41,083 44k INFO ====> Epoch: 4702, cost 14.29 s 2023-09-04 21:29:55,911 44k INFO ====> Epoch: 4703, cost 14.83 s 2023-09-04 21:30:10,871 44k INFO ====> Epoch: 4704, cost 14.96 s 2023-09-04 21:30:21,970 44k INFO Train Epoch: 4705 [71%] 2023-09-04 21:30:21,971 44k INFO Losses: [2.2115464210510254, 2.805337429046631, 9.166316032409668, 15.049665451049805, 0.6059744954109192], step: 98800, lr: 5.5541663506032095e-05, reference_loss: 29.838838577270508 2023-09-04 21:30:25,457 44k INFO ====> Epoch: 4705, cost 14.59 s 2023-09-04 21:30:39,875 44k INFO ====> Epoch: 4706, cost 14.42 s 2023-09-04 21:30:54,645 44k INFO ====> Epoch: 4707, cost 14.77 s 2023-09-04 21:31:09,430 44k INFO ====> Epoch: 4708, cost 14.79 s 2023-09-04 21:31:23,654 44k INFO ====> Epoch: 4709, cost 14.22 s 2023-09-04 21:31:37,782 44k INFO ====> Epoch: 4710, cost 14.13 s 2023-09-04 21:31:52,525 44k INFO ====> Epoch: 4711, cost 14.74 s 2023-09-04 21:32:07,219 44k INFO ====> Epoch: 4712, cost 14.69 s 2023-09-04 21:32:21,609 44k INFO ====> Epoch: 4713, cost 14.39 s 2023-09-04 21:32:36,058 44k INFO ====> Epoch: 4714, cost 14.45 s 2023-09-04 21:32:40,206 44k INFO Train Epoch: 4715 [24%] 2023-09-04 21:32:40,206 44k INFO Losses: [2.3726511001586914, 2.482891082763672, 9.176427841186523, 14.509737968444824, 0.5719790458679199], step: 99000, lr: 5.5472275466366955e-05, reference_loss: 29.11368751525879 2023-09-04 21:32:50,785 44k INFO ====> Epoch: 4715, cost 14.73 s 2023-09-04 21:33:05,285 44k INFO ====> Epoch: 4716, cost 14.50 s 2023-09-04 21:33:19,592 44k INFO ====> Epoch: 4717, cost 14.31 s 2023-09-04 21:33:33,855 44k INFO ====> Epoch: 4718, cost 14.26 s 2023-09-04 21:33:48,261 44k INFO ====> Epoch: 4719, cost 14.41 s 2023-09-04 21:34:02,814 44k INFO ====> Epoch: 4720, cost 14.55 s 2023-09-04 21:34:17,432 44k INFO ====> Epoch: 4721, cost 14.62 s 2023-09-04 21:34:32,009 44k INFO ====> Epoch: 4722, cost 14.58 s 2023-09-04 21:34:46,391 44k INFO ====> Epoch: 4723, cost 14.38 s 2023-09-04 21:34:58,504 44k INFO Train Epoch: 4724 [76%] 2023-09-04 21:34:58,504 44k INFO Losses: [2.328929901123047, 2.4509105682373047, 10.007888793945312, 17.861854553222656, 0.5696493983268738], step: 99200, lr: 5.5409900350523016e-05, reference_loss: 33.21923065185547 2023-09-04 21:35:04,011 44k INFO Saving model and optimizer state at iteration 4724 to ./logs\44k\G_99200.pth 2023-09-04 21:35:04,638 44k INFO Saving model and optimizer state at iteration 4724 to ./logs\44k\D_99200.pth 2023-09-04 21:35:06,366 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_96800.pth 2023-09-04 21:35:06,404 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_96800.pth 2023-09-04 21:35:08,980 44k INFO ====> Epoch: 4724, cost 22.59 s 2023-09-04 21:35:23,172 44k INFO ====> Epoch: 4725, cost 14.19 s 2023-09-04 21:35:37,475 44k INFO ====> Epoch: 4726, cost 14.30 s 2023-09-04 21:35:52,229 44k INFO ====> Epoch: 4727, cost 14.75 s 2023-09-04 21:36:07,161 44k INFO ====> Epoch: 4728, cost 14.93 s 2023-09-04 21:36:21,747 44k INFO ====> Epoch: 4729, cost 14.59 s 2023-09-04 21:36:36,279 44k INFO ====> Epoch: 4730, cost 14.53 s 2023-09-04 21:36:50,868 44k INFO ====> Epoch: 4731, cost 14.59 s 2023-09-04 21:37:05,500 44k INFO ====> Epoch: 4732, cost 14.63 s 2023-09-04 21:37:19,929 44k INFO ====> Epoch: 4733, cost 14.43 s 2023-09-04 21:37:24,755 44k INFO Train Epoch: 4734 [29%] 2023-09-04 21:37:24,755 44k INFO Losses: [2.254868507385254, 2.8135881423950195, 6.8311238288879395, 13.513243675231934, 0.5761390328407288], step: 99400, lr: 5.534067692218716e-05, reference_loss: 25.988962173461914 2023-09-04 21:37:34,286 44k INFO ====> Epoch: 4734, cost 14.36 s 2023-09-04 21:37:48,669 44k INFO ====> Epoch: 4735, cost 14.38 s 2023-09-04 21:38:03,275 44k INFO ====> Epoch: 4736, cost 14.61 s 2023-09-04 21:38:17,878 44k INFO ====> Epoch: 4737, cost 14.60 s 2023-09-04 21:38:32,351 44k INFO ====> Epoch: 4738, cost 14.47 s 2023-09-04 21:38:46,766 44k INFO ====> Epoch: 4739, cost 14.42 s 2023-09-04 21:39:01,384 44k INFO ====> Epoch: 4740, cost 14.62 s 2023-09-04 21:39:15,962 44k INFO ====> Epoch: 4741, cost 14.58 s 2023-09-04 21:39:30,076 44k INFO ====> Epoch: 4742, cost 14.11 s 2023-09-04 21:39:42,542 44k INFO Train Epoch: 4743 [81%] 2023-09-04 21:39:42,542 44k INFO Losses: [2.323673963546753, 2.647326946258545, 8.741546630859375, 15.635394096374512, 0.5546405911445618], step: 99600, lr: 5.5278449780702824e-05, reference_loss: 29.9025821685791 2023-09-04 21:39:44,667 44k INFO ====> Epoch: 4743, cost 14.59 s 2023-09-04 21:39:59,293 44k INFO ====> Epoch: 4744, cost 14.63 s 2023-09-04 21:40:13,888 44k INFO ====> Epoch: 4745, cost 14.60 s 2023-09-04 21:40:28,238 44k INFO ====> Epoch: 4746, cost 14.35 s 2023-09-04 21:40:42,662 44k INFO ====> Epoch: 4747, cost 14.42 s 2023-09-04 21:40:57,199 44k INFO ====> Epoch: 4748, cost 14.54 s 2023-09-04 21:41:11,987 44k INFO ====> Epoch: 4749, cost 14.79 s 2023-09-04 21:41:26,172 44k INFO ====> Epoch: 4750, cost 14.19 s 2023-09-04 21:41:40,313 44k INFO ====> Epoch: 4751, cost 14.14 s 2023-09-04 21:41:55,145 44k INFO ====> Epoch: 4752, cost 14.83 s 2023-09-04 21:42:01,022 44k INFO Train Epoch: 4753 [33%] 2023-09-04 21:42:01,022 44k INFO Losses: [2.3008298873901367, 2.5316193103790283, 7.585297584533691, 15.014857292175293, 0.5587667226791382], step: 99800, lr: 5.5209390573183875e-05, reference_loss: 27.991371154785156 2023-09-04 21:42:10,392 44k INFO ====> Epoch: 4753, cost 15.25 s 2023-09-04 21:42:24,784 44k INFO ====> Epoch: 4754, cost 14.39 s 2023-09-04 21:42:39,170 44k INFO ====> Epoch: 4755, cost 14.39 s 2023-09-04 21:42:53,914 44k INFO ====> Epoch: 4756, cost 14.74 s 2023-09-04 21:43:08,613 44k INFO ====> Epoch: 4757, cost 14.70 s 2023-09-04 21:43:22,866 44k INFO ====> Epoch: 4758, cost 14.25 s 2023-09-04 21:43:37,102 44k INFO ====> Epoch: 4759, cost 14.24 s 2023-09-04 21:43:51,836 44k INFO ====> Epoch: 4760, cost 14.73 s 2023-09-04 21:44:06,678 44k INFO ====> Epoch: 4761, cost 14.84 s 2023-09-04 21:44:20,126 44k INFO Train Epoch: 4762 [86%] 2023-09-04 21:44:20,126 44k INFO Losses: [2.225090980529785, 2.6451985836029053, 9.171111106872559, 16.63910484313965, 0.5808743238449097], step: 100000, lr: 5.514731105501513e-05, reference_loss: 31.26137924194336 2023-09-04 21:44:25,917 44k INFO Saving model and optimizer state at iteration 4762 to ./logs\44k\G_100000.pth 2023-09-04 21:44:26,621 44k INFO Saving model and optimizer state at iteration 4762 to ./logs\44k\D_100000.pth 2023-09-04 21:44:27,754 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_97600.pth 2023-09-04 21:44:27,796 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_97600.pth 2023-09-04 21:44:29,062 44k INFO ====> Epoch: 4762, cost 22.38 s 2023-09-04 21:44:44,506 44k INFO ====> Epoch: 4763, cost 15.44 s 2023-09-04 21:44:58,904 44k INFO ====> Epoch: 4764, cost 14.40 s 2023-09-04 21:45:13,529 44k INFO ====> Epoch: 4765, cost 14.62 s 2023-09-04 21:45:28,986 44k INFO ====> Epoch: 4766, cost 15.46 s 2023-09-04 21:45:43,985 44k INFO ====> Epoch: 4767, cost 15.00 s 2023-09-04 21:45:59,098 44k INFO ====> Epoch: 4768, cost 15.11 s 2023-09-04 21:46:13,917 44k INFO ====> Epoch: 4769, cost 14.82 s 2023-09-04 21:46:28,452 44k INFO ====> Epoch: 4770, cost 14.53 s 2023-09-04 21:46:43,018 44k INFO ====> Epoch: 4771, cost 14.57 s 2023-09-04 21:46:49,496 44k INFO Train Epoch: 4772 [38%] 2023-09-04 21:46:49,496 44k INFO Losses: [2.1357288360595703, 3.0179731845855713, 10.002586364746094, 16.015888214111328, 0.5566886067390442], step: 100200, lr: 5.50784156787271e-05, reference_loss: 31.728864669799805 2023-09-04 21:46:58,277 44k INFO ====> Epoch: 4772, cost 15.26 s 2023-09-04 21:47:13,223 44k INFO ====> Epoch: 4773, cost 14.95 s 2023-09-04 21:47:28,647 44k INFO ====> Epoch: 4774, cost 15.42 s 2023-09-04 21:47:43,554 44k INFO ====> Epoch: 4775, cost 14.91 s 2023-09-04 21:47:58,824 44k INFO ====> Epoch: 4776, cost 15.27 s 2023-09-04 21:48:13,845 44k INFO ====> Epoch: 4777, cost 15.02 s 2023-09-04 21:48:28,429 44k INFO ====> Epoch: 4778, cost 14.58 s 2023-09-04 21:48:43,224 44k INFO ====> Epoch: 4779, cost 14.80 s 2023-09-04 21:48:59,028 44k INFO ====> Epoch: 4780, cost 15.80 s 2023-09-04 21:49:15,121 44k INFO Train Epoch: 4781 [90%] 2023-09-04 21:49:15,122 44k INFO Losses: [2.1957666873931885, 2.782508134841919, 8.189508438110352, 15.226198196411133, 0.48188865184783936], step: 100400, lr: 5.5016483433662727e-05, reference_loss: 28.875869750976562 2023-09-04 21:49:15,880 44k INFO ====> Epoch: 4781, cost 16.85 s 2023-09-04 21:49:31,510 44k INFO ====> Epoch: 4782, cost 15.63 s 2023-09-04 21:49:47,003 44k INFO ====> Epoch: 4783, cost 15.49 s 2023-09-04 21:50:04,032 44k INFO ====> Epoch: 4784, cost 17.03 s 2023-09-04 21:50:19,392 44k INFO ====> Epoch: 4785, cost 15.36 s 2023-09-04 21:50:34,704 44k INFO ====> Epoch: 4786, cost 15.31 s 2023-09-04 21:50:49,772 44k INFO ====> Epoch: 4787, cost 15.07 s 2023-09-04 21:51:05,160 44k INFO ====> Epoch: 4788, cost 15.39 s 2023-09-04 21:51:20,968 44k INFO ====> Epoch: 4789, cost 15.81 s 2023-09-04 21:51:36,172 44k INFO ====> Epoch: 4790, cost 15.20 s 2023-09-04 21:51:43,709 44k INFO Train Epoch: 4791 [43%] 2023-09-04 21:51:43,709 44k INFO Losses: [2.1528468132019043, 2.7058420181274414, 8.951681137084961, 15.365291595458984, 0.5214750170707703], step: 100600, lr: 5.4947751499943865e-05, reference_loss: 29.69713592529297 2023-09-04 21:51:51,916 44k INFO ====> Epoch: 4791, cost 15.74 s 2023-09-04 21:52:07,549 44k INFO ====> Epoch: 4792, cost 15.63 s 2023-09-04 21:52:22,876 44k INFO ====> Epoch: 4793, cost 15.33 s 2023-09-04 21:52:38,217 44k INFO ====> Epoch: 4794, cost 15.34 s 2023-09-04 21:52:53,621 44k INFO ====> Epoch: 4795, cost 15.40 s 2023-09-04 21:53:09,584 44k INFO ====> Epoch: 4796, cost 15.96 s 2023-09-04 21:53:25,292 44k INFO ====> Epoch: 4797, cost 15.71 s 2023-09-04 21:53:40,841 44k INFO ====> Epoch: 4798, cost 15.55 s 2023-09-04 21:53:56,289 44k INFO ====> Epoch: 4799, cost 15.45 s 2023-09-04 21:54:11,588 44k INFO Train Epoch: 4800 [95%] 2023-09-04 21:54:11,589 44k INFO Losses: [2.1932363510131836, 2.4879982471466064, 10.270740509033203, 16.086637496948242, 0.305391401052475], step: 100800, lr: 5.4885966178603446e-05, reference_loss: 31.344003677368164 2023-09-04 21:54:17,493 44k INFO Saving model and optimizer state at iteration 4800 to ./logs\44k\G_100800.pth 2023-09-04 21:54:18,108 44k INFO Saving model and optimizer state at iteration 4800 to ./logs\44k\D_100800.pth 2023-09-04 21:54:19,322 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_98400.pth 2023-09-04 21:54:19,363 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_98400.pth 2023-09-04 21:54:19,364 44k INFO ====> Epoch: 4800, cost 23.07 s 2023-09-04 21:54:34,553 44k INFO ====> Epoch: 4801, cost 15.19 s 2023-09-04 21:54:49,899 44k INFO ====> Epoch: 4802, cost 15.35 s 2023-09-04 21:55:05,615 44k INFO ====> Epoch: 4803, cost 15.72 s 2023-09-04 21:55:21,061 44k INFO ====> Epoch: 4804, cost 15.45 s 2023-09-04 21:55:36,352 44k INFO ====> Epoch: 4805, cost 15.29 s 2023-09-04 21:55:51,364 44k INFO ====> Epoch: 4806, cost 15.01 s 2023-09-04 21:56:06,727 44k INFO ====> Epoch: 4807, cost 15.36 s 2023-09-04 21:56:22,542 44k INFO ====> Epoch: 4808, cost 15.82 s 2023-09-04 21:56:38,011 44k INFO ====> Epoch: 4809, cost 15.47 s 2023-09-04 21:56:46,295 44k INFO Train Epoch: 4810 [48%] 2023-09-04 21:56:46,295 44k INFO Losses: [2.2231359481811523, 2.6144180297851562, 8.9796781539917, 16.411468505859375, 0.706865668296814], step: 101000, lr: 5.481739729971406e-05, reference_loss: 30.935565948486328 2023-09-04 21:56:53,681 44k INFO ====> Epoch: 4810, cost 15.67 s 2023-09-04 21:57:09,485 44k INFO ====> Epoch: 4811, cost 15.80 s 2023-09-04 21:57:24,692 44k INFO ====> Epoch: 4812, cost 15.21 s 2023-09-04 21:57:40,065 44k INFO ====> Epoch: 4813, cost 15.37 s 2023-09-04 21:57:55,326 44k INFO ====> Epoch: 4814, cost 15.26 s 2023-09-04 21:58:10,675 44k INFO ====> Epoch: 4815, cost 15.35 s 2023-09-04 21:58:26,062 44k INFO ====> Epoch: 4816, cost 15.39 s 2023-09-04 21:58:41,153 44k INFO ====> Epoch: 4817, cost 15.09 s 2023-09-04 21:58:56,473 44k INFO ====> Epoch: 4818, cost 15.32 s 2023-09-04 21:59:11,576 44k INFO ====> Epoch: 4819, cost 15.10 s 2023-09-04 21:59:12,319 44k INFO Train Epoch: 4820 [0%] 2023-09-04 21:59:12,319 44k INFO Losses: [2.183115005493164, 2.8277599811553955, 8.238138198852539, 14.040665626525879, 0.6383637189865112], step: 101200, lr: 5.474891408372686e-05, reference_loss: 27.928041458129883 2023-09-04 21:59:26,939 44k INFO ====> Epoch: 4820, cost 15.36 s 2023-09-04 21:59:42,224 44k INFO ====> Epoch: 4821, cost 15.29 s 2023-09-04 21:59:57,387 44k INFO ====> Epoch: 4822, cost 15.16 s 2023-09-04 22:00:12,571 44k INFO ====> Epoch: 4823, cost 15.18 s 2023-09-04 22:00:27,890 44k INFO ====> Epoch: 4824, cost 15.32 s 2023-09-04 22:00:43,374 44k INFO ====> Epoch: 4825, cost 15.48 s 2023-09-04 22:00:58,895 44k INFO ====> Epoch: 4826, cost 15.52 s 2023-09-04 22:01:14,126 44k INFO ====> Epoch: 4827, cost 15.23 s 2023-09-04 22:01:29,692 44k INFO ====> Epoch: 4828, cost 15.57 s 2023-09-04 22:01:38,988 44k INFO Train Epoch: 4829 [52%] 2023-09-04 22:01:38,988 44k INFO Losses: [2.448953628540039, 2.4472153186798096, 7.793622970581055, 16.454221725463867, 0.4600990116596222], step: 101400, lr: 5.4687352342666253e-05, reference_loss: 29.604110717773438 2023-09-04 22:01:45,391 44k INFO ====> Epoch: 4829, cost 15.70 s 2023-09-04 22:02:00,615 44k INFO ====> Epoch: 4830, cost 15.22 s 2023-09-04 22:02:15,495 44k INFO ====> Epoch: 4831, cost 14.88 s 2023-09-04 22:02:30,309 44k INFO ====> Epoch: 4832, cost 14.81 s 2023-09-04 22:02:45,156 44k INFO ====> Epoch: 4833, cost 14.85 s 2023-09-04 22:02:59,864 44k INFO ====> Epoch: 4834, cost 14.71 s 2023-09-04 22:03:14,725 44k INFO ====> Epoch: 4835, cost 14.86 s 2023-09-04 22:03:29,747 44k INFO ====> Epoch: 4836, cost 15.02 s 2023-09-04 22:03:45,024 44k INFO ====> Epoch: 4837, cost 15.28 s 2023-09-04 22:04:00,190 44k INFO ====> Epoch: 4838, cost 15.17 s 2023-09-04 22:04:01,638 44k INFO Train Epoch: 4839 [5%] 2023-09-04 22:04:01,639 44k INFO Losses: [2.1665380001068115, 2.865673780441284, 9.589567184448242, 15.718674659729004, 0.5809982419013977], step: 101600, lr: 5.461903159146797e-05, reference_loss: 30.921451568603516 2023-09-04 22:04:07,391 44k INFO Saving model and optimizer state at iteration 4839 to ./logs\44k\G_101600.pth 2023-09-04 22:04:07,999 44k INFO Saving model and optimizer state at iteration 4839 to ./logs\44k\D_101600.pth 2023-09-04 22:04:09,179 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_99200.pth 2023-09-04 22:04:09,217 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_99200.pth 2023-09-04 22:04:22,439 44k INFO ====> Epoch: 4839, cost 22.25 s 2023-09-04 22:04:37,058 44k INFO ====> Epoch: 4840, cost 14.62 s 2023-09-04 22:04:51,706 44k INFO ====> Epoch: 4841, cost 14.65 s 2023-09-04 22:05:06,431 44k INFO ====> Epoch: 4842, cost 14.73 s 2023-09-04 22:05:21,129 44k INFO ====> Epoch: 4843, cost 14.70 s 2023-09-04 22:05:35,995 44k INFO ====> Epoch: 4844, cost 14.87 s 2023-09-04 22:05:51,027 44k INFO ====> Epoch: 4845, cost 15.03 s 2023-09-04 22:06:05,990 44k INFO ====> Epoch: 4846, cost 14.96 s 2023-09-04 22:06:20,462 44k INFO ====> Epoch: 4847, cost 14.47 s 2023-09-04 22:06:29,922 44k INFO Train Epoch: 4848 [57%] 2023-09-04 22:06:29,922 44k INFO Losses: [2.227088689804077, 2.6229071617126465, 7.2152533531188965, 14.014860153198242, 0.6182040572166443], step: 101800, lr: 5.455761589517355e-05, reference_loss: 26.698312759399414 2023-09-04 22:06:35,789 44k INFO ====> Epoch: 4848, cost 15.33 s 2023-09-04 22:06:50,323 44k INFO ====> Epoch: 4849, cost 14.53 s 2023-09-04 22:07:04,950 44k INFO ====> Epoch: 4850, cost 14.63 s 2023-09-04 22:07:19,432 44k INFO ====> Epoch: 4851, cost 14.48 s 2023-09-04 22:07:33,978 44k INFO ====> Epoch: 4852, cost 14.55 s 2023-09-04 22:07:48,565 44k INFO ====> Epoch: 4853, cost 14.59 s 2023-09-04 22:08:03,318 44k INFO ====> Epoch: 4854, cost 14.75 s 2023-09-04 22:08:17,862 44k INFO ====> Epoch: 4855, cost 14.54 s 2023-09-04 22:08:32,304 44k INFO ====> Epoch: 4856, cost 14.44 s 2023-09-04 22:08:47,251 44k INFO ====> Epoch: 4857, cost 14.95 s 2023-09-04 22:08:49,554 44k INFO Train Epoch: 4858 [10%] 2023-09-04 22:08:49,554 44k INFO Losses: [2.2369375228881836, 2.6336660385131836, 7.700793743133545, 14.822827339172363, 0.562576174736023], step: 102000, lr: 5.4489457223344106e-05, reference_loss: 27.956802368164062 2023-09-04 22:09:04,874 44k INFO ====> Epoch: 4858, cost 17.62 s 2023-09-04 22:09:20,509 44k INFO ====> Epoch: 4859, cost 15.63 s 2023-09-04 22:09:36,697 44k INFO ====> Epoch: 4860, cost 16.19 s 2023-09-04 22:09:53,953 44k INFO ====> Epoch: 4861, cost 17.26 s 2023-09-04 22:10:11,492 44k INFO ====> Epoch: 4862, cost 17.54 s 2023-09-04 22:10:26,892 44k INFO ====> Epoch: 4863, cost 15.40 s 2023-09-04 22:10:43,034 44k INFO ====> Epoch: 4864, cost 16.14 s 2023-09-04 22:10:58,164 44k INFO ====> Epoch: 4865, cost 15.13 s 2023-09-04 22:11:13,419 44k INFO ====> Epoch: 4866, cost 15.26 s 2023-09-04 22:11:23,728 44k INFO Train Epoch: 4867 [62%] 2023-09-04 22:11:23,729 44k INFO Losses: [2.1101062297821045, 2.8135557174682617, 10.227157592773438, 15.459447860717773, 0.5291330218315125], step: 102200, lr: 5.442818722534951e-05, reference_loss: 31.139400482177734 2023-09-04 22:11:28,897 44k INFO ====> Epoch: 4867, cost 15.48 s 2023-09-04 22:11:44,148 44k INFO ====> Epoch: 4868, cost 15.25 s 2023-09-04 22:11:59,935 44k INFO ====> Epoch: 4869, cost 15.79 s 2023-09-04 22:12:16,442 44k INFO ====> Epoch: 4870, cost 16.51 s 2023-09-04 22:12:31,603 44k INFO ====> Epoch: 4871, cost 15.16 s 2023-09-04 22:12:46,642 44k INFO ====> Epoch: 4872, cost 15.04 s 2023-09-04 22:13:01,822 44k INFO ====> Epoch: 4873, cost 15.18 s 2023-09-04 22:13:18,577 44k INFO ====> Epoch: 4874, cost 16.76 s 2023-09-04 22:13:33,827 44k INFO ====> Epoch: 4875, cost 15.25 s 2023-09-04 22:13:49,193 44k INFO ====> Epoch: 4876, cost 15.37 s 2023-09-04 22:13:52,111 44k INFO Train Epoch: 4877 [14%] 2023-09-04 22:13:52,111 44k INFO Losses: [2.2402801513671875, 2.6380274295806885, 9.596405982971191, 15.869318962097168, 0.6438015103340149], step: 102400, lr: 5.436019024838313e-05, reference_loss: 30.98783302307129 2023-09-04 22:13:57,994 44k INFO Saving model and optimizer state at iteration 4877 to ./logs\44k\G_102400.pth 2023-09-04 22:13:58,689 44k INFO Saving model and optimizer state at iteration 4877 to ./logs\44k\D_102400.pth 2023-09-04 22:13:59,819 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_100000.pth 2023-09-04 22:13:59,863 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_100000.pth 2023-09-04 22:14:12,646 44k INFO ====> Epoch: 4877, cost 23.45 s 2023-09-04 22:14:27,999 44k INFO ====> Epoch: 4878, cost 15.35 s 2023-09-04 22:14:43,397 44k INFO ====> Epoch: 4879, cost 15.40 s 2023-09-04 22:14:58,535 44k INFO ====> Epoch: 4880, cost 15.14 s 2023-09-04 22:15:14,071 44k INFO ====> Epoch: 4881, cost 15.54 s 2023-09-04 22:15:29,656 44k INFO ====> Epoch: 4882, cost 15.59 s 2023-09-04 22:15:45,240 44k INFO ====> Epoch: 4883, cost 15.58 s 2023-09-04 22:16:01,233 44k INFO ====> Epoch: 4884, cost 15.99 s 2023-09-04 22:16:17,031 44k INFO ====> Epoch: 4885, cost 15.80 s 2023-09-04 22:16:28,221 44k INFO Train Epoch: 4886 [67%] 2023-09-04 22:16:28,221 44k INFO Losses: [2.5849475860595703, 2.0504908561706543, 8.426408767700195, 14.882952690124512, 0.5818898677825928], step: 102600, lr: 5.42990656030439e-05, reference_loss: 28.526691436767578 2023-09-04 22:16:32,741 44k INFO ====> Epoch: 4886, cost 15.71 s 2023-09-04 22:16:48,225 44k INFO ====> Epoch: 4887, cost 15.48 s 2023-09-04 22:17:03,449 44k INFO ====> Epoch: 4888, cost 15.22 s 2023-09-04 22:17:18,646 44k INFO ====> Epoch: 4889, cost 15.20 s 2023-09-04 22:17:33,939 44k INFO ====> Epoch: 4890, cost 15.29 s 2023-09-04 22:17:49,111 44k INFO ====> Epoch: 4891, cost 15.17 s 2023-09-04 22:18:04,654 44k INFO ====> Epoch: 4892, cost 15.54 s 2023-09-04 22:18:19,831 44k INFO ====> Epoch: 4893, cost 15.18 s 2023-09-04 22:18:35,391 44k INFO ====> Epoch: 4894, cost 15.56 s 2023-09-04 22:18:50,685 44k INFO ====> Epoch: 4895, cost 15.29 s 2023-09-04 22:18:54,447 44k INFO Train Epoch: 4896 [19%] 2023-09-04 22:18:54,447 44k INFO Losses: [2.2658698558807373, 2.594343900680542, 9.538492202758789, 16.09035873413086, 0.5601411461830139], step: 102800, lr: 5.423122993734702e-05, reference_loss: 31.049205780029297 2023-09-04 22:19:06,389 44k INFO ====> Epoch: 4896, cost 15.70 s 2023-09-04 22:19:21,736 44k INFO ====> Epoch: 4897, cost 15.35 s 2023-09-04 22:19:37,304 44k INFO ====> Epoch: 4898, cost 15.57 s 2023-09-04 22:19:51,911 44k INFO ====> Epoch: 4899, cost 14.61 s 2023-09-04 22:20:07,114 44k INFO ====> Epoch: 4900, cost 15.20 s 2023-09-04 22:20:21,742 44k INFO ====> Epoch: 4901, cost 14.63 s 2023-09-04 22:20:36,369 44k INFO ====> Epoch: 4902, cost 14.63 s 2023-09-04 22:20:51,375 44k INFO ====> Epoch: 4903, cost 15.01 s 2023-09-04 22:21:06,463 44k INFO ====> Epoch: 4904, cost 15.09 s 2023-09-04 22:21:17,849 44k INFO Train Epoch: 4905 [71%] 2023-09-04 22:21:17,850 44k INFO Losses: [2.0698323249816895, 2.7569212913513184, 9.932278633117676, 14.550992012023926, 0.426900178194046], step: 103000, lr: 5.4170250299838674e-05, reference_loss: 29.73692512512207 2023-09-04 22:21:21,318 44k INFO ====> Epoch: 4905, cost 14.86 s 2023-09-04 22:21:36,814 44k INFO ====> Epoch: 4906, cost 15.50 s 2023-09-04 22:21:52,324 44k INFO ====> Epoch: 4907, cost 15.51 s 2023-09-04 22:22:07,557 44k INFO ====> Epoch: 4908, cost 15.23 s 2023-09-04 22:22:23,456 44k INFO ====> Epoch: 4909, cost 15.90 s 2023-09-04 22:22:40,191 44k INFO ====> Epoch: 4910, cost 16.74 s 2023-09-04 22:22:54,983 44k INFO ====> Epoch: 4911, cost 14.79 s 2023-09-04 22:23:10,655 44k INFO ====> Epoch: 4912, cost 15.67 s 2023-09-04 22:23:28,054 44k INFO ====> Epoch: 4913, cost 17.40 s 2023-09-04 22:23:43,086 44k INFO ====> Epoch: 4914, cost 15.03 s 2023-09-04 22:23:47,521 44k INFO Train Epoch: 4915 [24%] 2023-09-04 22:23:47,521 44k INFO Losses: [2.310577154159546, 2.460390090942383, 8.202776908874512, 15.584057807922363, 0.665442705154419], step: 103200, lr: 5.410257556272772e-05, reference_loss: 29.22324562072754 2023-09-04 22:23:53,341 44k INFO Saving model and optimizer state at iteration 4915 to ./logs\44k\G_103200.pth 2023-09-04 22:23:54,074 44k INFO Saving model and optimizer state at iteration 4915 to ./logs\44k\D_103200.pth 2023-09-04 22:23:55,699 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_100800.pth 2023-09-04 22:23:55,740 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_100800.pth 2023-09-04 22:24:07,350 44k INFO ====> Epoch: 4915, cost 24.26 s 2023-09-04 22:24:24,119 44k INFO ====> Epoch: 4916, cost 16.77 s 2023-09-04 22:24:40,718 44k INFO ====> Epoch: 4917, cost 16.60 s 2023-09-04 22:24:57,020 44k INFO ====> Epoch: 4918, cost 16.30 s 2023-09-04 22:25:12,874 44k INFO ====> Epoch: 4919, cost 15.86 s 2023-09-04 22:25:29,135 44k INFO ====> Epoch: 4920, cost 16.26 s 2023-09-04 22:25:44,617 44k INFO ====> Epoch: 4921, cost 15.48 s 2023-09-04 22:26:02,135 44k INFO ====> Epoch: 4922, cost 17.52 s 2023-09-04 22:26:18,989 44k INFO ====> Epoch: 4923, cost 16.85 s 2023-09-04 22:26:32,733 44k INFO Train Epoch: 4924 [76%] 2023-09-04 22:26:32,733 44k INFO Losses: [2.1008098125457764, 2.6067004203796387, 10.773721694946289, 17.629426956176758, 0.7346259355545044], step: 103400, lr: 5.404174058904384e-05, reference_loss: 33.84528350830078 2023-09-04 22:26:36,193 44k INFO ====> Epoch: 4924, cost 17.20 s 2023-09-04 22:26:52,458 44k INFO ====> Epoch: 4925, cost 16.27 s 2023-09-04 22:27:07,691 44k INFO ====> Epoch: 4926, cost 15.23 s 2023-09-04 22:27:22,963 44k INFO ====> Epoch: 4927, cost 15.27 s 2023-09-04 22:27:39,078 44k INFO ====> Epoch: 4928, cost 16.12 s 2023-09-04 22:27:54,669 44k INFO ====> Epoch: 4929, cost 15.59 s 2023-09-04 22:28:10,459 44k INFO ====> Epoch: 4930, cost 15.79 s 2023-09-04 22:28:26,328 44k INFO ====> Epoch: 4931, cost 15.87 s 2023-09-04 22:28:42,077 44k INFO ====> Epoch: 4932, cost 15.75 s 2023-09-04 22:28:57,529 44k INFO ====> Epoch: 4933, cost 15.45 s 2023-09-04 22:29:03,035 44k INFO Train Epoch: 4934 [29%] 2023-09-04 22:29:03,036 44k INFO Losses: [2.058730125427246, 2.777829170227051, 8.852826118469238, 16.074363708496094, 0.551540732383728], step: 103600, lr: 5.397422639874311e-05, reference_loss: 30.315288543701172 2023-09-04 22:29:14,362 44k INFO ====> Epoch: 4934, cost 16.83 s 2023-09-04 22:29:31,881 44k INFO ====> Epoch: 4935, cost 17.52 s 2023-09-04 22:30:03,447 44k INFO ====> Epoch: 4936, cost 31.57 s 2023-09-04 22:30:38,404 44k INFO ====> Epoch: 4937, cost 34.96 s 2023-09-04 22:31:12,416 44k INFO ====> Epoch: 4938, cost 34.01 s 2023-09-04 22:31:57,095 44k INFO ====> Epoch: 4939, cost 44.68 s 2023-09-04 22:32:37,300 44k INFO ====> Epoch: 4940, cost 40.20 s 2023-09-04 22:33:13,950 44k INFO ====> Epoch: 4941, cost 36.65 s 2023-09-04 22:33:50,065 44k INFO ====> Epoch: 4942, cost 36.12 s 2023-09-04 22:34:20,581 44k INFO Train Epoch: 4943 [81%] 2023-09-04 22:34:20,603 44k INFO Losses: [2.082242965698242, 2.837239980697632, 9.465642929077148, 15.450313568115234, 0.6845446825027466], step: 103800, lr: 5.391353574569337e-05, reference_loss: 30.51998519897461 2023-09-04 22:34:25,805 44k INFO ====> Epoch: 4943, cost 35.74 s 2023-09-04 22:34:51,214 44k INFO ====> Epoch: 4944, cost 25.41 s 2023-09-04 22:35:14,963 44k INFO ====> Epoch: 4945, cost 23.75 s 2023-09-04 22:35:59,032 44k INFO ====> Epoch: 4946, cost 44.07 s 2023-09-04 22:36:40,370 44k INFO ====> Epoch: 4947, cost 41.34 s 2023-09-04 22:37:53,981 44k INFO ====> Epoch: 4948, cost 73.61 s 2023-09-04 22:38:36,191 44k INFO ====> Epoch: 4949, cost 42.21 s 2023-09-04 22:39:18,238 44k INFO ====> Epoch: 4950, cost 42.05 s 2023-09-04 22:40:00,570 44k INFO ====> Epoch: 4951, cost 42.33 s 2023-09-04 22:40:42,598 44k INFO ====> Epoch: 4952, cost 42.03 s 2023-09-04 22:40:56,391 44k INFO Train Epoch: 4953 [33%] 2023-09-04 22:40:56,434 44k INFO Losses: [2.521454334259033, 2.300662040710449, 7.146838188171387, 15.384712219238281, 0.6193235516548157], step: 104000, lr: 5.384618172133284e-05, reference_loss: 27.972990036010742 2023-09-04 22:41:04,833 44k INFO Saving model and optimizer state at iteration 4953 to ./logs\44k\G_104000.pth 2023-09-04 22:41:06,208 44k INFO Saving model and optimizer state at iteration 4953 to ./logs\44k\D_104000.pth 2023-09-04 22:41:07,402 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_101600.pth 2023-09-04 22:41:07,466 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_101600.pth 2023-09-04 22:41:32,235 44k INFO ====> Epoch: 4953, cost 49.64 s 2023-09-04 22:42:09,107 44k INFO ====> Epoch: 4954, cost 36.87 s 2023-09-04 22:42:49,676 44k INFO ====> Epoch: 4955, cost 40.57 s 2023-09-04 22:43:28,807 44k INFO ====> Epoch: 4956, cost 39.13 s 2023-09-04 22:44:08,044 44k INFO ====> Epoch: 4957, cost 39.24 s 2023-09-04 22:44:49,195 44k INFO ====> Epoch: 4958, cost 41.15 s 2023-09-04 22:45:29,203 44k INFO ====> Epoch: 4959, cost 40.01 s 2023-09-04 22:46:09,344 44k INFO ====> Epoch: 4960, cost 40.14 s 2023-09-04 22:46:50,835 44k INFO ====> Epoch: 4961, cost 41.49 s 2023-09-04 22:47:29,400 44k INFO Train Epoch: 4962 [86%] 2023-09-04 22:47:29,428 44k INFO Losses: [2.3339054584503174, 2.7221901416778564, 10.269947052001953, 15.700551986694336, 0.7104182839393616], step: 104200, lr: 5.378563504654105e-05, reference_loss: 31.73701286315918 2023-09-04 22:47:33,022 44k INFO ====> Epoch: 4962, cost 42.19 s 2023-09-04 22:48:14,745 44k INFO ====> Epoch: 4963, cost 41.72 s 2023-09-04 22:48:57,517 44k INFO ====> Epoch: 4964, cost 42.77 s 2023-09-04 22:49:38,369 44k INFO ====> Epoch: 4965, cost 40.85 s 2023-09-04 22:50:19,372 44k INFO ====> Epoch: 4966, cost 41.00 s 2023-09-04 22:50:59,795 44k INFO ====> Epoch: 4967, cost 40.42 s 2023-09-04 22:51:40,392 44k INFO ====> Epoch: 4968, cost 40.60 s 2023-09-04 22:52:20,933 44k INFO ====> Epoch: 4969, cost 40.54 s 2023-09-04 22:53:00,199 44k INFO ====> Epoch: 4970, cost 39.27 s 2023-09-04 22:53:22,545 44k INFO ====> Epoch: 4971, cost 22.35 s 2023-09-04 22:53:36,768 44k INFO Train Epoch: 4972 [38%] 2023-09-04 22:53:36,774 44k INFO Losses: [2.059579610824585, 2.8397905826568604, 8.885653495788574, 15.543844223022461, 0.5280100107192993], step: 104400, lr: 5.371844080815425e-05, reference_loss: 29.85687828063965 2023-09-04 22:53:58,061 44k INFO ====> Epoch: 4972, cost 35.52 s 2023-09-04 22:54:35,836 44k INFO ====> Epoch: 4973, cost 37.78 s 2023-09-04 22:55:24,793 44k INFO ====> Epoch: 4974, cost 48.96 s 2023-09-04 22:56:04,428 44k INFO ====> Epoch: 4975, cost 39.64 s 2023-09-04 22:57:01,283 44k INFO ====> Epoch: 4976, cost 56.85 s 2023-09-04 22:57:58,243 44k INFO ====> Epoch: 4977, cost 56.96 s 2023-09-04 22:58:56,934 44k INFO ====> Epoch: 4978, cost 58.69 s 2023-09-04 22:59:43,206 44k INFO ====> Epoch: 4979, cost 46.27 s 2023-09-04 23:00:22,880 44k INFO ====> Epoch: 4980, cost 39.67 s 2023-09-04 23:01:01,586 44k INFO Train Epoch: 4981 [90%] 2023-09-04 23:01:01,640 44k INFO Losses: [2.245450258255005, 2.641481876373291, 9.423460960388184, 16.98377227783203, 0.42979511618614197], step: 104600, lr: 5.3658037770056485e-05, reference_loss: 31.723960876464844 2023-09-04 23:01:03,950 44k INFO ====> Epoch: 4981, cost 41.07 s 2023-09-04 23:01:50,817 44k INFO ====> Epoch: 4982, cost 46.87 s 2023-09-04 23:02:30,734 44k INFO ====> Epoch: 4983, cost 39.92 s 2023-09-04 23:03:10,469 44k INFO ====> Epoch: 4984, cost 39.74 s 2023-09-04 23:03:47,783 44k INFO ====> Epoch: 4985, cost 37.31 s 2023-09-04 23:04:36,907 44k INFO ====> Epoch: 4986, cost 49.12 s 2023-09-04 23:05:06,918 44k INFO ====> Epoch: 4987, cost 30.01 s 2023-09-04 23:06:22,495 44k INFO ====> Epoch: 4988, cost 75.58 s 2023-09-04 23:07:30,869 44k INFO ====> Epoch: 4989, cost 68.37 s 2023-09-04 23:08:36,411 44k INFO ====> Epoch: 4990, cost 65.54 s 2023-09-04 23:09:23,737 44k INFO Train Epoch: 4991 [43%] 2023-09-04 23:09:23,763 44k INFO Losses: [2.4455976486206055, 2.2915728092193604, 8.03713607788086, 13.179948806762695, 0.7167665362358093], step: 104800, lr: 5.3591002938578345e-05, reference_loss: 26.6710205078125 2023-09-04 23:09:33,281 44k INFO Saving model and optimizer state at iteration 4991 to ./logs\44k\G_104800.pth 2023-09-04 23:09:34,070 44k INFO Saving model and optimizer state at iteration 4991 to ./logs\44k\D_104800.pth 2023-09-04 23:09:35,257 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_102400.pth 2023-09-04 23:09:35,257 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_102400.pth 2023-09-04 23:10:18,722 44k INFO ====> Epoch: 4991, cost 102.31 s 2023-09-04 23:11:25,898 44k INFO ====> Epoch: 4992, cost 67.18 s 2023-09-04 23:13:02,745 44k INFO ====> Epoch: 4993, cost 96.85 s 2023-09-04 23:15:12,268 44k INFO ====> Epoch: 4994, cost 129.52 s 2023-09-04 23:16:25,327 44k INFO ====> Epoch: 4995, cost 73.06 s 2023-09-04 23:17:08,661 44k INFO ====> Epoch: 4996, cost 43.33 s 2023-09-04 23:17:50,517 44k INFO ====> Epoch: 4997, cost 41.86 s 2023-09-04 23:18:47,413 44k INFO ====> Epoch: 4998, cost 56.90 s 2023-09-04 23:20:45,423 44k INFO ====> Epoch: 4999, cost 118.01 s 2023-09-04 23:21:29,615 44k INFO Train Epoch: 5000 [95%] 2023-09-04 23:21:29,625 44k INFO Losses: [2.170355796813965, 2.704669713973999, 7.864683151245117, 15.209817886352539, 0.5708844661712646], step: 105000, lr: 5.3530743196420954e-05, reference_loss: 28.52041244506836 2023-09-04 23:21:29,992 44k INFO ====> Epoch: 5000, cost 44.57 s 2023-09-04 23:22:21,132 44k INFO ====> Epoch: 5001, cost 51.14 s 2023-09-04 23:23:16,540 44k INFO ====> Epoch: 5002, cost 55.41 s 2023-09-04 23:24:01,840 44k INFO ====> Epoch: 5003, cost 45.30 s 2023-09-04 23:24:56,537 44k INFO ====> Epoch: 5004, cost 54.70 s 2023-09-04 23:25:49,338 44k INFO ====> Epoch: 5005, cost 52.80 s 2023-09-04 23:26:39,879 44k INFO ====> Epoch: 5006, cost 50.54 s 2023-09-04 23:27:22,919 44k INFO ====> Epoch: 5007, cost 43.04 s 2023-09-04 23:28:04,956 44k INFO ====> Epoch: 5008, cost 42.04 s 2023-09-04 23:28:47,353 44k INFO ====> Epoch: 5009, cost 42.40 s 2023-09-04 23:29:10,025 44k INFO Train Epoch: 5010 [48%] 2023-09-04 23:29:10,067 44k INFO Losses: [2.377894878387451, 2.51542329788208, 9.741415023803711, 17.12163543701172, 0.6142744421958923], step: 105200, lr: 5.3463867393685685e-05, reference_loss: 32.370643615722656 2023-09-04 23:29:30,671 44k INFO ====> Epoch: 5010, cost 43.32 s 2023-09-04 23:30:07,753 44k INFO ====> Epoch: 5011, cost 37.08 s 2023-09-04 23:30:25,005 44k INFO ====> Epoch: 5012, cost 17.25 s 2023-09-04 23:30:41,872 44k INFO ====> Epoch: 5013, cost 16.87 s 2023-09-04 23:30:58,637 44k INFO ====> Epoch: 5014, cost 16.76 s 2023-09-04 23:31:14,902 44k INFO ====> Epoch: 5015, cost 16.26 s 2023-09-04 23:31:32,509 44k INFO ====> Epoch: 5016, cost 17.61 s 2023-09-04 23:31:51,569 44k INFO ====> Epoch: 5017, cost 19.06 s 2023-09-04 23:32:14,744 44k INFO ====> Epoch: 5018, cost 23.18 s 2023-09-04 23:32:57,880 44k INFO ====> Epoch: 5019, cost 43.14 s 2023-09-04 23:33:00,315 44k INFO Train Epoch: 5020 [0%] 2023-09-04 23:33:00,332 44k INFO Losses: [2.2054672241210938, 3.057281017303467, 9.114274024963379, 14.218461036682129, 0.676436185836792], step: 105400, lr: 5.339707513869745e-05, reference_loss: 29.27191734313965 2023-09-04 23:33:37,407 44k INFO ====> Epoch: 5020, cost 39.53 s 2023-09-04 23:34:30,285 44k INFO ====> Epoch: 5021, cost 52.88 s 2023-09-04 23:35:06,823 44k INFO ====> Epoch: 5022, cost 36.54 s 2023-09-04 23:35:46,936 44k INFO ====> Epoch: 5023, cost 40.11 s 2023-09-04 23:36:37,528 44k INFO ====> Epoch: 5024, cost 50.59 s 2023-09-04 23:38:00,551 44k INFO ====> Epoch: 5025, cost 83.02 s 2023-09-04 23:38:50,895 44k INFO ====> Epoch: 5026, cost 50.34 s 2023-09-04 23:39:28,413 44k INFO ====> Epoch: 5027, cost 37.52 s 2023-09-04 23:40:03,566 44k INFO ====> Epoch: 5028, cost 35.15 s 2023-09-04 23:41:33,529 44k INFO Train Epoch: 5029 [52%] 2023-09-04 23:41:33,621 44k INFO Losses: [2.101332187652588, 3.0443217754364014, 9.405316352844238, 16.207923889160156, 0.5883114337921143], step: 105600, lr: 5.333703345626235e-05, reference_loss: 31.347206115722656 2023-09-04 23:41:45,277 44k INFO Saving model and optimizer state at iteration 5029 to ./logs\44k\G_105600.pth 2023-09-04 23:41:46,122 44k INFO Saving model and optimizer state at iteration 5029 to ./logs\44k\D_105600.pth 2023-09-04 23:41:47,379 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_103200.pth 2023-09-04 23:41:47,379 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_103200.pth 2023-09-04 23:42:11,076 44k INFO ====> Epoch: 5029, cost 127.51 s 2023-09-04 23:43:36,039 44k INFO ====> Epoch: 5030, cost 84.96 s 2023-09-04 23:44:24,064 44k INFO ====> Epoch: 5031, cost 48.03 s 2023-09-04 23:45:07,230 44k INFO ====> Epoch: 5032, cost 43.17 s 2023-09-04 23:46:29,279 44k INFO ====> Epoch: 5033, cost 82.05 s 2023-09-04 23:47:15,468 44k INFO ====> Epoch: 5034, cost 46.19 s 2023-09-04 23:47:56,251 44k INFO ====> Epoch: 5035, cost 40.78 s 2023-09-04 23:48:13,317 44k INFO ====> Epoch: 5036, cost 17.07 s 2023-09-04 23:48:29,994 44k INFO ====> Epoch: 5037, cost 16.68 s 2023-09-04 23:48:46,317 44k INFO ====> Epoch: 5038, cost 16.32 s 2023-09-04 23:48:47,879 44k INFO Train Epoch: 5039 [5%] 2023-09-04 23:48:47,879 44k INFO Losses: [2.392210006713867, 2.420614004135132, 7.78708028793335, 13.648435592651367, 0.6443769335746765], step: 105800, lr: 5.327039965454552e-05, reference_loss: 26.892717361450195 2023-09-04 23:49:02,298 44k INFO ====> Epoch: 5039, cost 15.98 s 2023-09-04 23:49:19,164 44k INFO ====> Epoch: 5040, cost 16.87 s 2023-09-04 23:49:36,118 44k INFO ====> Epoch: 5041, cost 16.95 s 2023-09-04 23:49:52,948 44k INFO ====> Epoch: 5042, cost 16.83 s 2023-09-04 23:50:10,581 44k INFO ====> Epoch: 5043, cost 17.63 s 2023-09-04 23:50:27,508 44k INFO ====> Epoch: 5044, cost 16.93 s 2023-09-04 23:50:44,753 44k INFO ====> Epoch: 5045, cost 17.24 s 2023-09-04 23:51:01,140 44k INFO ====> Epoch: 5046, cost 16.39 s 2023-09-04 23:51:17,241 44k INFO ====> Epoch: 5047, cost 16.10 s 2023-09-04 23:51:27,897 44k INFO Train Epoch: 5048 [57%] 2023-09-04 23:51:27,898 44k INFO Losses: [2.3729653358459473, 2.364046812057495, 8.823601722717285, 16.336633682250977, 0.4918058514595032], step: 106000, lr: 5.3210500410795904e-05, reference_loss: 30.389053344726562 2023-09-04 23:51:34,511 44k INFO ====> Epoch: 5048, cost 17.27 s 2023-09-04 23:51:51,209 44k INFO ====> Epoch: 5049, cost 16.70 s 2023-09-04 23:52:07,226 44k INFO ====> Epoch: 5050, cost 16.02 s 2023-09-04 23:52:23,664 44k INFO ====> Epoch: 5051, cost 16.44 s 2023-09-04 23:52:40,325 44k INFO ====> Epoch: 5052, cost 16.66 s 2023-09-04 23:52:56,527 44k INFO ====> Epoch: 5053, cost 16.20 s 2023-09-04 23:53:13,129 44k INFO ====> Epoch: 5054, cost 16.60 s 2023-09-04 23:53:29,882 44k INFO ====> Epoch: 5055, cost 16.75 s 2023-09-04 23:53:46,269 44k INFO ====> Epoch: 5056, cost 16.39 s 2023-09-04 23:54:02,597 44k INFO ====> Epoch: 5057, cost 16.33 s 2023-09-04 23:54:04,983 44k INFO Train Epoch: 5058 [10%] 2023-09-04 23:54:04,984 44k INFO Losses: [2.0571882724761963, 3.0909318923950195, 8.775802612304688, 14.931522369384766, 0.6445450186729431], step: 106200, lr: 5.3144024686447004e-05, reference_loss: 29.499990463256836 2023-09-04 23:54:19,354 44k INFO ====> Epoch: 5058, cost 16.76 s 2023-09-04 23:54:35,784 44k INFO ====> Epoch: 5059, cost 16.43 s 2023-09-04 23:54:52,300 44k INFO ====> Epoch: 5060, cost 16.52 s 2023-09-04 23:55:08,634 44k INFO ====> Epoch: 5061, cost 16.33 s 2023-09-04 23:55:25,324 44k INFO ====> Epoch: 5062, cost 16.69 s 2023-09-04 23:55:42,010 44k INFO ====> Epoch: 5063, cost 16.69 s 2023-09-04 23:55:58,418 44k INFO ====> Epoch: 5064, cost 16.41 s 2023-09-04 23:56:14,879 44k INFO ====> Epoch: 5065, cost 16.46 s 2023-09-04 23:56:31,289 44k INFO ====> Epoch: 5066, cost 16.41 s 2023-09-04 23:56:42,564 44k INFO Train Epoch: 5067 [62%] 2023-09-04 23:56:42,564 44k INFO Losses: [2.435642719268799, 2.322408676147461, 8.585687637329102, 14.730015754699707, 0.5884018540382385], step: 106400, lr: 5.308426754347131e-05, reference_loss: 28.662155151367188 2023-09-04 23:56:48,684 44k INFO Saving model and optimizer state at iteration 5067 to ./logs\44k\G_106400.pth 2023-09-04 23:56:49,390 44k INFO Saving model and optimizer state at iteration 5067 to ./logs\44k\D_106400.pth 2023-09-04 23:56:50,516 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_104000.pth 2023-09-04 23:56:50,516 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_104000.pth 2023-09-04 23:56:55,849 44k INFO ====> Epoch: 5067, cost 24.56 s 2023-09-04 23:57:12,409 44k INFO ====> Epoch: 5068, cost 16.56 s 2023-09-04 23:57:28,824 44k INFO ====> Epoch: 5069, cost 16.42 s 2023-09-04 23:57:45,344 44k INFO ====> Epoch: 5070, cost 16.52 s 2023-09-04 23:58:01,972 44k INFO ====> Epoch: 5071, cost 16.63 s 2023-09-04 23:58:18,149 44k INFO ====> Epoch: 5072, cost 16.18 s 2023-09-04 23:58:34,383 44k INFO ====> Epoch: 5073, cost 16.23 s 2023-09-04 23:58:50,835 44k INFO ====> Epoch: 5074, cost 16.45 s 2023-09-04 23:59:06,722 44k INFO ====> Epoch: 5075, cost 15.89 s 2023-09-04 23:59:22,660 44k INFO ====> Epoch: 5076, cost 15.94 s 2023-09-04 23:59:25,735 44k INFO Train Epoch: 5077 [14%] 2023-09-04 23:59:25,735 44k INFO Losses: [2.3884406089782715, 2.452091932296753, 9.091933250427246, 16.7452449798584, 0.6739130020141602], step: 106600, lr: 5.301794952147866e-05, reference_loss: 31.35162353515625 2023-09-04 23:59:38,665 44k INFO ====> Epoch: 5077, cost 16.00 s 2023-09-04 23:59:54,595 44k INFO ====> Epoch: 5078, cost 15.93 s 2023-09-05 00:00:11,262 44k INFO ====> Epoch: 5079, cost 16.67 s 2023-09-05 00:00:27,782 44k INFO ====> Epoch: 5080, cost 16.52 s 2023-09-05 00:00:44,440 44k INFO ====> Epoch: 5081, cost 16.66 s 2023-09-05 00:01:01,071 44k INFO ====> Epoch: 5082, cost 16.63 s 2023-09-05 00:01:17,606 44k INFO ====> Epoch: 5083, cost 16.54 s 2023-09-05 00:01:34,391 44k INFO ====> Epoch: 5084, cost 16.78 s 2023-09-05 00:01:51,021 44k INFO ====> Epoch: 5085, cost 16.63 s 2023-09-05 00:02:02,982 44k INFO Train Epoch: 5086 [67%] 2023-09-05 00:02:02,982 44k INFO Losses: [2.094825506210327, 2.9468560218811035, 10.275693893432617, 16.086936950683594, 0.5164368152618408], step: 106800, lr: 5.2958334142166956e-05, reference_loss: 31.92074966430664 2023-09-05 00:02:07,665 44k INFO ====> Epoch: 5086, cost 16.64 s 2023-09-05 00:02:24,001 44k INFO ====> Epoch: 5087, cost 16.34 s 2023-09-05 00:02:40,334 44k INFO ====> Epoch: 5088, cost 16.33 s 2023-09-05 00:02:56,942 44k INFO ====> Epoch: 5089, cost 16.61 s 2023-09-05 00:03:13,179 44k INFO ====> Epoch: 5090, cost 16.24 s 2023-09-05 00:03:29,364 44k INFO ====> Epoch: 5091, cost 16.19 s 2023-09-05 00:03:45,855 44k INFO ====> Epoch: 5092, cost 16.49 s 2023-09-05 00:04:02,487 44k INFO ====> Epoch: 5093, cost 16.63 s 2023-09-05 00:04:18,910 44k INFO ====> Epoch: 5094, cost 16.42 s 2023-09-05 00:04:35,443 44k INFO ====> Epoch: 5095, cost 16.53 s 2023-09-05 00:04:39,377 44k INFO Train Epoch: 5096 [19%] 2023-09-05 00:04:39,377 44k INFO Losses: [2.2310192584991455, 2.59814715385437, 7.503283500671387, 14.914413452148438, 0.47461891174316406], step: 107000, lr: 5.289217344840853e-05, reference_loss: 27.72148323059082 2023-09-05 00:04:52,223 44k INFO ====> Epoch: 5096, cost 16.78 s 2023-09-05 00:05:08,712 44k INFO ====> Epoch: 5097, cost 16.49 s 2023-09-05 00:05:25,145 44k INFO ====> Epoch: 5098, cost 16.43 s 2023-09-05 00:05:41,590 44k INFO ====> Epoch: 5099, cost 16.44 s 2023-09-05 00:05:57,611 44k INFO ====> Epoch: 5100, cost 16.02 s 2023-09-05 00:06:13,909 44k INFO ====> Epoch: 5101, cost 16.30 s 2023-09-05 00:06:30,253 44k INFO ====> Epoch: 5102, cost 16.34 s 2023-09-05 00:06:46,263 44k INFO ====> Epoch: 5103, cost 16.01 s 2023-09-05 00:07:02,752 44k INFO ====> Epoch: 5104, cost 16.49 s 2023-09-05 00:07:15,382 44k INFO Train Epoch: 5105 [71%] 2023-09-05 00:07:15,382 44k INFO Losses: [2.239246368408203, 2.6818597316741943, 8.925917625427246, 15.323896408081055, 0.5912581086158752], step: 107200, lr: 5.2832699496450615e-05, reference_loss: 29.76218032836914 2023-09-05 00:07:21,394 44k INFO Saving model and optimizer state at iteration 5105 to ./logs\44k\G_107200.pth 2023-09-05 00:07:22,046 44k INFO Saving model and optimizer state at iteration 5105 to ./logs\44k\D_107200.pth 2023-09-05 00:07:23,218 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_104800.pth 2023-09-05 00:07:23,252 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_104800.pth 2023-09-05 00:07:26,980 44k INFO ====> Epoch: 5105, cost 24.23 s 2023-09-05 00:07:43,180 44k INFO ====> Epoch: 5106, cost 16.20 s 2023-09-05 00:07:58,985 44k INFO ====> Epoch: 5107, cost 15.80 s 2023-09-05 00:08:14,796 44k INFO ====> Epoch: 5108, cost 15.81 s 2023-09-05 00:08:31,173 44k INFO ====> Epoch: 5109, cost 16.38 s 2023-09-05 00:08:47,818 44k INFO ====> Epoch: 5110, cost 16.64 s 2023-09-05 00:09:04,324 44k INFO ====> Epoch: 5111, cost 16.51 s 2023-09-05 00:09:20,772 44k INFO ====> Epoch: 5112, cost 16.45 s 2023-09-05 00:09:37,266 44k INFO ====> Epoch: 5113, cost 16.49 s 2023-09-05 00:09:53,461 44k INFO ====> Epoch: 5114, cost 16.19 s 2023-09-05 00:09:58,064 44k INFO Train Epoch: 5115 [24%] 2023-09-05 00:09:58,065 44k INFO Losses: [2.199841022491455, 2.719879388809204, 7.3010759353637695, 14.818282127380371, 0.6282963752746582], step: 107400, lr: 5.276669575769191e-05, reference_loss: 27.667373657226562 2023-09-05 00:10:09,864 44k INFO ====> Epoch: 5115, cost 16.40 s 2023-09-05 00:10:26,441 44k INFO ====> Epoch: 5116, cost 16.58 s 2023-09-05 00:10:42,981 44k INFO ====> Epoch: 5117, cost 16.54 s 2023-09-05 00:10:59,078 44k INFO ====> Epoch: 5118, cost 16.10 s 2023-09-05 00:11:15,308 44k INFO ====> Epoch: 5119, cost 16.23 s 2023-09-05 00:11:31,639 44k INFO ====> Epoch: 5120, cost 16.33 s 2023-09-05 00:11:48,018 44k INFO ====> Epoch: 5121, cost 16.38 s 2023-09-05 00:12:04,140 44k INFO ====> Epoch: 5122, cost 16.12 s 2023-09-05 00:12:20,419 44k INFO ====> Epoch: 5123, cost 16.28 s 2023-09-05 00:12:33,744 44k INFO Train Epoch: 5124 [76%] 2023-09-05 00:12:33,744 44k INFO Losses: [2.1989266872406006, 2.6948800086975098, 9.304139137268066, 16.778703689575195, 0.4978886544704437], step: 107600, lr: 5.270736289757543e-05, reference_loss: 31.474538803100586 2023-09-05 00:12:36,865 44k INFO ====> Epoch: 5124, cost 16.45 s 2023-09-05 00:12:53,037 44k INFO ====> Epoch: 5125, cost 16.17 s 2023-09-05 00:13:09,148 44k INFO ====> Epoch: 5126, cost 16.11 s 2023-09-05 00:13:24,431 44k INFO ====> Epoch: 5127, cost 15.28 s 2023-09-05 00:13:39,293 44k INFO ====> Epoch: 5128, cost 14.86 s 2023-09-05 00:13:55,181 44k INFO ====> Epoch: 5129, cost 15.89 s 2023-09-05 00:14:11,141 44k INFO ====> Epoch: 5130, cost 15.96 s 2023-09-05 00:14:26,636 44k INFO ====> Epoch: 5131, cost 15.50 s 2023-09-05 00:14:42,133 44k INFO ====> Epoch: 5132, cost 15.50 s 2023-09-05 00:14:57,418 44k INFO ====> Epoch: 5133, cost 15.28 s 2023-09-05 00:15:02,545 44k INFO Train Epoch: 5134 [29%] 2023-09-05 00:15:02,546 44k INFO Losses: [2.1454038619995117, 2.6396117210388184, 8.922677040100098, 13.130949020385742, 0.4515206217765808], step: 107800, lr: 5.264151574146739e-05, reference_loss: 27.290163040161133 2023-09-05 00:15:12,733 44k INFO ====> Epoch: 5134, cost 15.32 s 2023-09-05 00:15:28,263 44k INFO ====> Epoch: 5135, cost 15.53 s 2023-09-05 00:15:43,644 44k INFO ====> Epoch: 5136, cost 15.38 s 2023-09-05 00:15:59,377 44k INFO ====> Epoch: 5137, cost 15.73 s 2023-09-05 00:16:15,384 44k INFO ====> Epoch: 5138, cost 16.01 s 2023-09-05 00:16:30,587 44k INFO ====> Epoch: 5139, cost 15.20 s 2023-09-05 00:16:46,032 44k INFO ====> Epoch: 5140, cost 15.44 s 2023-09-05 00:17:01,275 44k INFO ====> Epoch: 5141, cost 15.24 s 2023-09-05 00:17:16,767 44k INFO ====> Epoch: 5142, cost 15.49 s 2023-09-05 00:17:30,205 44k INFO Train Epoch: 5143 [81%] 2023-09-05 00:17:30,207 44k INFO Losses: [2.1374802589416504, 2.92557954788208, 9.470208168029785, 15.014388084411621, 0.6008853316307068], step: 108000, lr: 5.258232363847595e-05, reference_loss: 30.148542404174805 2023-09-05 00:17:36,185 44k INFO Saving model and optimizer state at iteration 5143 to ./logs\44k\G_108000.pth 2023-09-05 00:17:36,876 44k INFO Saving model and optimizer state at iteration 5143 to ./logs\44k\D_108000.pth 2023-09-05 00:17:37,988 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_105600.pth 2023-09-05 00:17:38,031 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_105600.pth 2023-09-05 00:17:39,998 44k INFO ====> Epoch: 5143, cost 23.23 s 2023-09-05 00:17:55,336 44k INFO ====> Epoch: 5144, cost 15.34 s 2023-09-05 00:18:10,580 44k INFO ====> Epoch: 5145, cost 15.24 s 2023-09-05 00:18:26,113 44k INFO ====> Epoch: 5146, cost 15.53 s 2023-09-05 00:18:41,485 44k INFO ====> Epoch: 5147, cost 15.37 s 2023-09-05 00:18:56,792 44k INFO ====> Epoch: 5148, cost 15.31 s 2023-09-05 00:19:12,069 44k INFO ====> Epoch: 5149, cost 15.28 s 2023-09-05 00:19:27,443 44k INFO ====> Epoch: 5150, cost 15.37 s 2023-09-05 00:19:42,713 44k INFO ====> Epoch: 5151, cost 15.27 s 2023-09-05 00:19:58,079 44k INFO ====> Epoch: 5152, cost 15.37 s 2023-09-05 00:20:04,106 44k INFO Train Epoch: 5153 [33%] 2023-09-05 00:20:04,106 44k INFO Losses: [2.3169515132904053, 2.5430824756622314, 7.976562976837158, 13.954283714294434, 0.6123846173286438], step: 108200, lr: 5.251663269355286e-05, reference_loss: 27.40326690673828 2023-09-05 00:20:13,845 44k INFO ====> Epoch: 5153, cost 15.77 s 2023-09-05 00:20:29,062 44k INFO ====> Epoch: 5154, cost 15.22 s 2023-09-05 00:20:44,211 44k INFO ====> Epoch: 5155, cost 15.15 s 2023-09-05 00:20:59,354 44k INFO ====> Epoch: 5156, cost 15.14 s 2023-09-05 00:21:14,670 44k INFO ====> Epoch: 5157, cost 15.32 s 2023-09-05 00:21:30,074 44k INFO ====> Epoch: 5158, cost 15.40 s 2023-09-05 00:21:45,447 44k INFO ====> Epoch: 5159, cost 15.37 s 2023-09-05 00:22:00,755 44k INFO ====> Epoch: 5160, cost 15.31 s 2023-09-05 00:22:16,029 44k INFO ====> Epoch: 5161, cost 15.27 s 2023-09-05 00:22:30,318 44k INFO Train Epoch: 5162 [86%] 2023-09-05 00:22:30,318 44k INFO Losses: [2.2641496658325195, 2.5410118103027344, 8.435314178466797, 16.244159698486328, 0.5431413650512695], step: 108400, lr: 5.2457581013764094e-05, reference_loss: 30.027774810791016 2023-09-05 00:22:31,754 44k INFO ====> Epoch: 5162, cost 15.73 s 2023-09-05 00:22:47,122 44k INFO ====> Epoch: 5163, cost 15.37 s 2023-09-05 00:23:02,571 44k INFO ====> Epoch: 5164, cost 15.45 s 2023-09-05 00:23:18,109 44k INFO ====> Epoch: 5165, cost 15.54 s 2023-09-05 00:23:33,656 44k INFO ====> Epoch: 5166, cost 15.55 s 2023-09-05 00:23:49,110 44k INFO ====> Epoch: 5167, cost 15.46 s 2023-09-05 00:24:04,226 44k INFO ====> Epoch: 5168, cost 15.12 s 2023-09-05 00:24:19,592 44k INFO ====> Epoch: 5169, cost 15.37 s 2023-09-05 00:24:34,922 44k INFO ====> Epoch: 5170, cost 15.33 s 2023-09-05 00:24:50,209 44k INFO ====> Epoch: 5171, cost 15.29 s 2023-09-05 00:24:56,726 44k INFO Train Epoch: 5172 [38%] 2023-09-05 00:24:56,727 44k INFO Losses: [2.1066672801971436, 2.9201619625091553, 9.514408111572266, 15.518081665039062, 0.4496857225894928], step: 108600, lr: 5.239204590944146e-05, reference_loss: 30.509004592895508 2023-09-05 00:25:05,680 44k INFO ====> Epoch: 5172, cost 15.47 s 2023-09-05 00:25:20,943 44k INFO ====> Epoch: 5173, cost 15.26 s 2023-09-05 00:25:36,418 44k INFO ====> Epoch: 5174, cost 15.47 s 2023-09-05 00:25:51,707 44k INFO ====> Epoch: 5175, cost 15.29 s 2023-09-05 00:26:07,235 44k INFO ====> Epoch: 5176, cost 15.53 s 2023-09-05 00:26:22,462 44k INFO ====> Epoch: 5177, cost 15.23 s 2023-09-05 00:26:37,594 44k INFO ====> Epoch: 5178, cost 15.13 s 2023-09-05 00:26:53,263 44k INFO ====> Epoch: 5179, cost 15.67 s 2023-09-05 00:27:08,657 44k INFO ====> Epoch: 5180, cost 15.39 s 2023-09-05 00:27:23,715 44k INFO Train Epoch: 5181 [90%] 2023-09-05 00:27:23,715 44k INFO Losses: [2.326486825942993, 2.6887729167938232, 7.704583168029785, 15.063507080078125, 0.690622866153717], step: 108800, lr: 5.2333134319725185e-05, reference_loss: 28.47397232055664 2023-09-05 00:27:29,596 44k INFO Saving model and optimizer state at iteration 5181 to ./logs\44k\G_108800.pth 2023-09-05 00:27:30,283 44k INFO Saving model and optimizer state at iteration 5181 to ./logs\44k\D_108800.pth 2023-09-05 00:27:31,460 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_106400.pth 2023-09-05 00:27:31,499 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_106400.pth 2023-09-05 00:27:31,979 44k INFO ====> Epoch: 5181, cost 23.32 s 2023-09-05 00:27:47,639 44k INFO ====> Epoch: 5182, cost 15.66 s 2023-09-05 00:28:02,872 44k INFO ====> Epoch: 5183, cost 15.23 s 2023-09-05 00:28:18,161 44k INFO ====> Epoch: 5184, cost 15.29 s 2023-09-05 00:28:33,271 44k INFO ====> Epoch: 5185, cost 15.11 s 2023-09-05 00:28:48,980 44k INFO ====> Epoch: 5186, cost 15.71 s 2023-09-05 00:29:04,334 44k INFO ====> Epoch: 5187, cost 15.35 s 2023-09-05 00:29:19,531 44k INFO ====> Epoch: 5188, cost 15.20 s 2023-09-05 00:29:35,013 44k INFO ====> Epoch: 5189, cost 15.48 s 2023-09-05 00:29:50,567 44k INFO ====> Epoch: 5190, cost 15.55 s 2023-09-05 00:29:58,245 44k INFO Train Epoch: 5191 [43%] 2023-09-05 00:29:58,246 44k INFO Losses: [2.320068836212158, 2.4276840686798096, 8.586791038513184, 14.69007682800293, 0.5518888235092163], step: 109000, lr: 5.226775468629768e-05, reference_loss: 28.576509475708008 2023-09-05 00:30:06,527 44k INFO ====> Epoch: 5191, cost 15.96 s 2023-09-05 00:30:21,935 44k INFO ====> Epoch: 5192, cost 15.41 s 2023-09-05 00:30:37,096 44k INFO ====> Epoch: 5193, cost 15.16 s 2023-09-05 00:30:52,954 44k INFO ====> Epoch: 5194, cost 15.86 s 2023-09-05 00:31:08,472 44k INFO ====> Epoch: 5195, cost 15.52 s 2023-09-05 00:31:23,863 44k INFO ====> Epoch: 5196, cost 15.39 s 2023-09-05 00:31:39,191 44k INFO ====> Epoch: 5197, cost 15.33 s 2023-09-05 00:31:54,276 44k INFO ====> Epoch: 5198, cost 15.09 s 2023-09-05 00:32:09,660 44k INFO ====> Epoch: 5199, cost 15.38 s 2023-09-05 00:32:25,039 44k INFO Train Epoch: 5200 [95%] 2023-09-05 00:32:25,040 44k INFO Losses: [2.0157203674316406, 2.9604246616363525, 12.137486457824707, 16.443328857421875, 0.24198687076568604], step: 109200, lr: 5.2208982854314014e-05, reference_loss: 33.798946380615234 2023-09-05 00:32:25,374 44k INFO ====> Epoch: 5200, cost 15.71 s 2023-09-05 00:32:40,658 44k INFO ====> Epoch: 5201, cost 15.28 s 2023-09-05 00:32:55,564 44k INFO ====> Epoch: 5202, cost 14.91 s 2023-09-05 00:33:11,297 44k INFO ====> Epoch: 5203, cost 15.73 s 2023-09-05 00:33:27,073 44k INFO ====> Epoch: 5204, cost 15.78 s 2023-09-05 00:33:42,361 44k INFO ====> Epoch: 5205, cost 15.29 s 2023-09-05 00:33:57,408 44k INFO ====> Epoch: 5206, cost 15.05 s 2023-09-05 00:34:12,602 44k INFO ====> Epoch: 5207, cost 15.19 s 2023-09-05 00:34:27,567 44k INFO ====> Epoch: 5208, cost 14.96 s 2023-09-05 00:34:42,702 44k INFO ====> Epoch: 5209, cost 15.14 s 2023-09-05 00:34:50,653 44k INFO Train Epoch: 5210 [48%] 2023-09-05 00:34:50,653 44k INFO Losses: [2.4134531021118164, 2.4317054748535156, 8.290695190429688, 15.714317321777344, 0.5046124458312988], step: 109400, lr: 5.2143758322953373e-05, reference_loss: 29.354782104492188 2023-09-05 00:34:58,047 44k INFO ====> Epoch: 5210, cost 15.35 s 2023-09-05 00:35:13,460 44k INFO ====> Epoch: 5211, cost 15.41 s 2023-09-05 00:35:28,932 44k INFO ====> Epoch: 5212, cost 15.47 s 2023-09-05 00:35:44,028 44k INFO ====> Epoch: 5213, cost 15.10 s 2023-09-05 00:35:59,282 44k INFO ====> Epoch: 5214, cost 15.25 s 2023-09-05 00:36:14,649 44k INFO ====> Epoch: 5215, cost 15.37 s 2023-09-05 00:36:29,785 44k INFO ====> Epoch: 5216, cost 15.14 s 2023-09-05 00:36:44,794 44k INFO ====> Epoch: 5217, cost 15.01 s 2023-09-05 00:37:00,105 44k INFO ====> Epoch: 5218, cost 15.31 s 2023-09-05 00:37:14,704 44k INFO ====> Epoch: 5219, cost 14.60 s 2023-09-05 00:37:15,432 44k INFO Train Epoch: 5220 [0%] 2023-09-05 00:37:15,432 44k INFO Losses: [2.0945262908935547, 2.8529164791107178, 9.02824592590332, 14.897088050842285, 0.5776115655899048], step: 109600, lr: 5.2078615276411205e-05, reference_loss: 29.450387954711914 2023-09-05 00:37:21,039 44k INFO Saving model and optimizer state at iteration 5220 to ./logs\44k\G_109600.pth 2023-09-05 00:37:21,730 44k INFO Saving model and optimizer state at iteration 5220 to ./logs\44k\D_109600.pth 2023-09-05 00:37:22,956 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_107200.pth 2023-09-05 00:37:22,986 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_107200.pth 2023-09-05 00:37:37,034 44k INFO ====> Epoch: 5220, cost 22.33 s 2023-09-05 00:37:51,659 44k INFO ====> Epoch: 5221, cost 14.62 s 2023-09-05 00:38:06,520 44k INFO ====> Epoch: 5222, cost 14.86 s 2023-09-05 00:38:20,965 44k INFO ====> Epoch: 5223, cost 14.44 s 2023-09-05 00:38:35,698 44k INFO ====> Epoch: 5224, cost 14.73 s 2023-09-05 00:38:51,082 44k INFO ====> Epoch: 5225, cost 15.38 s 2023-09-05 00:39:05,655 44k INFO ====> Epoch: 5226, cost 14.57 s 2023-09-05 00:39:20,896 44k INFO ====> Epoch: 5227, cost 15.24 s 2023-09-05 00:39:36,188 44k INFO ====> Epoch: 5228, cost 15.29 s 2023-09-05 00:39:44,935 44k INFO Train Epoch: 5229 [52%] 2023-09-05 00:39:44,935 44k INFO Losses: [2.087036371231079, 2.9189953804016113, 8.228788375854492, 15.906625747680664, 0.621610164642334], step: 109800, lr: 5.2020056119903775e-05, reference_loss: 29.7630558013916 2023-09-05 00:39:51,444 44k INFO ====> Epoch: 5229, cost 15.26 s 2023-09-05 00:40:06,333 44k INFO ====> Epoch: 5230, cost 14.89 s 2023-09-05 00:40:21,581 44k INFO ====> Epoch: 5231, cost 15.25 s 2023-09-05 00:40:36,823 44k INFO ====> Epoch: 5232, cost 15.24 s 2023-09-05 00:40:51,937 44k INFO ====> Epoch: 5233, cost 15.11 s 2023-09-05 00:41:07,100 44k INFO ====> Epoch: 5234, cost 15.16 s 2023-09-05 00:41:23,557 44k INFO ====> Epoch: 5235, cost 16.46 s 2023-09-05 00:41:38,511 44k INFO ====> Epoch: 5236, cost 14.95 s 2023-09-05 00:41:53,386 44k INFO ====> Epoch: 5237, cost 14.88 s 2023-09-05 00:42:08,232 44k INFO ====> Epoch: 5238, cost 14.85 s 2023-09-05 00:42:09,664 44k INFO Train Epoch: 5239 [5%] 2023-09-05 00:42:09,664 44k INFO Losses: [2.3311028480529785, 2.798100471496582, 10.169112205505371, 16.04230499267578, 0.5006618499755859], step: 110000, lr: 5.1955067614166314e-05, reference_loss: 31.84128189086914 2023-09-05 00:42:23,170 44k INFO ====> Epoch: 5239, cost 14.94 s 2023-09-05 00:42:37,876 44k INFO ====> Epoch: 5240, cost 14.71 s 2023-09-05 00:42:52,894 44k INFO ====> Epoch: 5241, cost 15.02 s 2023-09-05 00:43:07,708 44k INFO ====> Epoch: 5242, cost 14.81 s 2023-09-05 00:43:22,524 44k INFO ====> Epoch: 5243, cost 14.82 s 2023-09-05 00:43:37,293 44k INFO ====> Epoch: 5244, cost 14.77 s 2023-09-05 00:43:52,224 44k INFO ====> Epoch: 5245, cost 14.93 s 2023-09-05 00:44:07,312 44k INFO ====> Epoch: 5246, cost 15.09 s 2023-09-05 00:44:22,190 44k INFO ====> Epoch: 5247, cost 14.88 s 2023-09-05 00:44:31,417 44k INFO Train Epoch: 5248 [57%] 2023-09-05 00:44:31,417 44k INFO Losses: [2.386312961578369, 2.548947811126709, 7.932812690734863, 16.123666763305664, 0.4769439697265625], step: 110200, lr: 5.189664737930361e-05, reference_loss: 29.468685150146484 2023-09-05 00:44:37,059 44k INFO ====> Epoch: 5248, cost 14.87 s 2023-09-05 00:44:51,912 44k INFO ====> Epoch: 5249, cost 14.85 s 2023-09-05 00:45:07,042 44k INFO ====> Epoch: 5250, cost 15.13 s 2023-09-05 00:45:21,897 44k INFO ====> Epoch: 5251, cost 14.85 s 2023-09-05 00:45:37,861 44k INFO ====> Epoch: 5252, cost 15.96 s 2023-09-05 00:45:52,912 44k INFO ====> Epoch: 5253, cost 15.05 s 2023-09-05 00:46:07,601 44k INFO ====> Epoch: 5254, cost 14.69 s 2023-09-05 00:46:22,359 44k INFO ====> Epoch: 5255, cost 14.76 s 2023-09-05 00:46:36,971 44k INFO ====> Epoch: 5256, cost 14.61 s 2023-09-05 00:46:51,827 44k INFO ====> Epoch: 5257, cost 14.86 s 2023-09-05 00:46:54,020 44k INFO Train Epoch: 5258 [10%] 2023-09-05 00:46:54,021 44k INFO Losses: [2.434117317199707, 2.139406204223633, 6.808326244354248, 14.567167282104492, 0.6857479810714722], step: 110400, lr: 5.183181304774903e-05, reference_loss: 26.634763717651367 2023-09-05 00:46:59,795 44k INFO Saving model and optimizer state at iteration 5258 to ./logs\44k\G_110400.pth 2023-09-05 00:47:00,380 44k INFO Saving model and optimizer state at iteration 5258 to ./logs\44k\D_110400.pth 2023-09-05 00:47:01,553 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_108000.pth 2023-09-05 00:47:01,591 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_108000.pth 2023-09-05 00:47:14,191 44k INFO ====> Epoch: 5258, cost 22.36 s 2023-09-05 00:47:28,708 44k INFO ====> Epoch: 5259, cost 14.52 s 2023-09-05 00:47:43,188 44k INFO ====> Epoch: 5260, cost 14.48 s 2023-09-05 00:47:57,801 44k INFO ====> Epoch: 5261, cost 14.61 s 2023-09-05 00:48:12,539 44k INFO ====> Epoch: 5262, cost 14.74 s 2023-09-05 00:48:27,002 44k INFO ====> Epoch: 5263, cost 14.46 s 2023-09-05 00:48:41,476 44k INFO ====> Epoch: 5264, cost 14.47 s 2023-09-05 00:48:56,059 44k INFO ====> Epoch: 5265, cost 14.58 s 2023-09-05 00:49:10,685 44k INFO ====> Epoch: 5266, cost 14.63 s 2023-09-05 00:49:20,439 44k INFO Train Epoch: 5267 [62%] 2023-09-05 00:49:20,439 44k INFO Losses: [2.3673062324523926, 2.359941244125366, 9.528892517089844, 16.148197174072266, 0.4776000678539276], step: 110600, lr: 5.177353140496307e-05, reference_loss: 30.88193702697754 2023-09-05 00:49:25,412 44k INFO ====> Epoch: 5267, cost 14.73 s 2023-09-05 00:49:39,822 44k INFO ====> Epoch: 5268, cost 14.41 s 2023-09-05 00:49:54,407 44k INFO ====> Epoch: 5269, cost 14.59 s 2023-09-05 00:50:08,989 44k INFO ====> Epoch: 5270, cost 14.58 s 2023-09-05 00:50:23,684 44k INFO ====> Epoch: 5271, cost 14.70 s 2023-09-05 00:50:38,177 44k INFO ====> Epoch: 5272, cost 14.49 s 2023-09-05 00:50:52,856 44k INFO ====> Epoch: 5273, cost 14.68 s 2023-09-05 00:51:07,454 44k INFO ====> Epoch: 5274, cost 14.60 s 2023-09-05 00:51:21,933 44k INFO ====> Epoch: 5275, cost 14.48 s 2023-09-05 00:51:36,476 44k INFO ====> Epoch: 5276, cost 14.54 s 2023-09-05 00:51:39,331 44k INFO Train Epoch: 5277 [14%] 2023-09-05 00:51:39,331 44k INFO Losses: [2.102895498275757, 2.822896957397461, 8.574385643005371, 16.610740661621094, 0.5422603487968445], step: 110800, lr: 5.170885088183935e-05, reference_loss: 30.653181076049805 2023-09-05 00:51:51,589 44k INFO ====> Epoch: 5277, cost 15.11 s 2023-09-05 00:52:06,321 44k INFO ====> Epoch: 5278, cost 14.73 s 2023-09-05 00:52:20,979 44k INFO ====> Epoch: 5279, cost 14.66 s 2023-09-05 00:52:35,599 44k INFO ====> Epoch: 5280, cost 14.62 s 2023-09-05 00:52:50,391 44k INFO ====> Epoch: 5281, cost 14.79 s 2023-09-05 00:53:05,251 44k INFO ====> Epoch: 5282, cost 14.86 s 2023-09-05 00:53:19,789 44k INFO ====> Epoch: 5283, cost 14.54 s 2023-09-05 00:53:34,266 44k INFO ====> Epoch: 5284, cost 14.48 s 2023-09-05 00:53:48,792 44k INFO ====> Epoch: 5285, cost 14.53 s 2023-09-05 00:53:59,573 44k INFO Train Epoch: 5286 [67%] 2023-09-05 00:53:59,574 44k INFO Losses: [2.256621837615967, 2.7017338275909424, 7.3196001052856445, 13.984130859375, 0.48126187920570374], step: 111000, lr: 5.1650707502343985e-05, reference_loss: 26.743349075317383 2023-09-05 00:54:03,857 44k INFO ====> Epoch: 5286, cost 15.06 s 2023-09-05 00:54:18,426 44k INFO ====> Epoch: 5287, cost 14.57 s 2023-09-05 00:54:33,026 44k INFO ====> Epoch: 5288, cost 14.60 s 2023-09-05 00:54:47,501 44k INFO ====> Epoch: 5289, cost 14.48 s 2023-09-05 00:55:02,092 44k INFO ====> Epoch: 5290, cost 14.59 s 2023-09-05 00:55:16,752 44k INFO ====> Epoch: 5291, cost 14.66 s 2023-09-05 00:55:31,399 44k INFO ====> Epoch: 5292, cost 14.65 s 2023-09-05 00:55:46,654 44k INFO ====> Epoch: 5293, cost 15.25 s 2023-09-05 00:56:01,453 44k INFO ====> Epoch: 5294, cost 14.80 s 2023-09-05 00:56:15,992 44k INFO ====> Epoch: 5295, cost 14.54 s 2023-09-05 00:56:19,487 44k INFO Train Epoch: 5296 [19%] 2023-09-05 00:56:19,488 44k INFO Losses: [2.328579902648926, 2.489997386932373, 8.150876998901367, 14.605923652648926, 0.6624788045883179], step: 111200, lr: 5.158618042276676e-05, reference_loss: 28.237857818603516 2023-09-05 00:56:25,028 44k INFO Saving model and optimizer state at iteration 5296 to ./logs\44k\G_111200.pth 2023-09-05 00:56:25,663 44k INFO Saving model and optimizer state at iteration 5296 to ./logs\44k\D_111200.pth 2023-09-05 00:56:26,834 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_108800.pth 2023-09-05 00:56:26,875 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_108800.pth 2023-09-05 00:56:38,024 44k INFO ====> Epoch: 5296, cost 22.03 s 2023-09-05 00:56:52,405 44k INFO ====> Epoch: 5297, cost 14.38 s 2023-09-05 00:57:06,868 44k INFO ====> Epoch: 5298, cost 14.46 s 2023-09-05 00:57:21,379 44k INFO ====> Epoch: 5299, cost 14.51 s 2023-09-05 00:57:35,898 44k INFO ====> Epoch: 5300, cost 14.52 s 2023-09-05 00:57:50,446 44k INFO ====> Epoch: 5301, cost 14.55 s 2023-09-05 00:58:05,238 44k INFO ====> Epoch: 5302, cost 14.79 s 2023-09-05 00:58:19,874 44k INFO ====> Epoch: 5303, cost 14.64 s 2023-09-05 00:58:34,634 44k INFO ====> Epoch: 5304, cost 14.76 s 2023-09-05 00:58:45,988 44k INFO Train Epoch: 5305 [71%] 2023-09-05 00:58:45,988 44k INFO Losses: [2.274402141571045, 2.666853427886963, 8.461166381835938, 14.6131010055542, 0.7198076844215393], step: 111400, lr: 5.152817497855584e-05, reference_loss: 28.73533058166504 2023-09-05 00:58:49,547 44k INFO ====> Epoch: 5305, cost 14.91 s 2023-09-05 00:59:04,176 44k INFO ====> Epoch: 5306, cost 14.63 s 2023-09-05 00:59:18,941 44k INFO ====> Epoch: 5307, cost 14.76 s 2023-09-05 00:59:33,475 44k INFO ====> Epoch: 5308, cost 14.53 s 2023-09-05 00:59:48,154 44k INFO ====> Epoch: 5309, cost 14.68 s 2023-09-05 01:00:03,168 44k INFO ====> Epoch: 5310, cost 15.01 s 2023-09-05 01:00:17,777 44k INFO ====> Epoch: 5311, cost 14.61 s 2023-09-05 01:00:32,159 44k INFO ====> Epoch: 5312, cost 14.38 s 2023-09-05 01:00:46,678 44k INFO ====> Epoch: 5313, cost 14.52 s 2023-09-05 01:01:01,219 44k INFO ====> Epoch: 5314, cost 14.54 s 2023-09-05 01:01:05,471 44k INFO Train Epoch: 5315 [24%] 2023-09-05 01:01:05,471 44k INFO Losses: [2.3437209129333496, 2.5839860439300537, 7.217243194580078, 15.847607612609863, 0.4530520737171173], step: 111600, lr: 5.146380097850639e-05, reference_loss: 28.44561004638672 2023-09-05 01:01:16,062 44k INFO ====> Epoch: 5315, cost 14.84 s 2023-09-05 01:01:30,393 44k INFO ====> Epoch: 5316, cost 14.33 s 2023-09-05 01:01:44,874 44k INFO ====> Epoch: 5317, cost 14.48 s 2023-09-05 01:01:59,544 44k INFO ====> Epoch: 5318, cost 14.67 s 2023-09-05 01:02:14,165 44k INFO ====> Epoch: 5319, cost 14.62 s 2023-09-05 01:02:28,653 44k INFO ====> Epoch: 5320, cost 14.49 s 2023-09-05 01:02:43,277 44k INFO ====> Epoch: 5321, cost 14.62 s 2023-09-05 01:02:57,674 44k INFO ====> Epoch: 5322, cost 14.40 s 2023-09-05 01:03:12,159 44k INFO ====> Epoch: 5323, cost 14.48 s 2023-09-05 01:03:24,011 44k INFO Train Epoch: 5324 [76%] 2023-09-05 01:03:24,011 44k INFO Losses: [2.4436068534851074, 2.172537088394165, 7.665647029876709, 13.344378471374512, 0.49340441823005676], step: 111800, lr: 5.140593314235192e-05, reference_loss: 26.11957550048828 2023-09-05 01:03:26,815 44k INFO ====> Epoch: 5324, cost 14.66 s 2023-09-05 01:03:41,324 44k INFO ====> Epoch: 5325, cost 14.51 s 2023-09-05 01:03:55,774 44k INFO ====> Epoch: 5326, cost 14.45 s 2023-09-05 01:04:10,429 44k INFO ====> Epoch: 5327, cost 14.66 s 2023-09-05 01:04:24,933 44k INFO ====> Epoch: 5328, cost 14.50 s 2023-09-05 01:04:39,325 44k INFO ====> Epoch: 5329, cost 14.39 s 2023-09-05 01:04:54,044 44k INFO ====> Epoch: 5330, cost 14.72 s 2023-09-05 01:05:08,664 44k INFO ====> Epoch: 5331, cost 14.62 s 2023-09-05 01:05:23,222 44k INFO ====> Epoch: 5332, cost 14.56 s 2023-09-05 01:05:37,554 44k INFO ====> Epoch: 5333, cost 14.33 s 2023-09-05 01:05:42,493 44k INFO Train Epoch: 5334 [29%] 2023-09-05 01:05:42,494 44k INFO Losses: [2.142242193222046, 2.6875007152557373, 10.652972221374512, 17.23539161682129, 0.6697046160697937], step: 112000, lr: 5.1341711858675076e-05, reference_loss: 33.387813568115234 2023-09-05 01:05:47,899 44k INFO Saving model and optimizer state at iteration 5334 to ./logs\44k\G_112000.pth 2023-09-05 01:05:48,543 44k INFO Saving model and optimizer state at iteration 5334 to ./logs\44k\D_112000.pth 2023-09-05 01:05:49,778 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_109600.pth 2023-09-05 01:05:49,816 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_109600.pth 2023-09-05 01:05:59,538 44k INFO ====> Epoch: 5334, cost 21.98 s 2023-09-05 01:06:14,753 44k INFO ====> Epoch: 5335, cost 15.21 s 2023-09-05 01:06:29,263 44k INFO ====> Epoch: 5336, cost 14.51 s 2023-09-05 01:06:43,795 44k INFO ====> Epoch: 5337, cost 14.53 s 2023-09-05 01:06:58,493 44k INFO ====> Epoch: 5338, cost 14.70 s 2023-09-05 01:07:12,988 44k INFO ====> Epoch: 5339, cost 14.49 s 2023-09-05 01:07:27,508 44k INFO ====> Epoch: 5340, cost 14.52 s 2023-09-05 01:07:42,094 44k INFO ====> Epoch: 5341, cost 14.59 s 2023-09-05 01:07:56,778 44k INFO ====> Epoch: 5342, cost 14.68 s 2023-09-05 01:08:09,692 44k INFO Train Epoch: 5343 [81%] 2023-09-05 01:08:09,692 44k INFO Losses: [2.2159647941589355, 2.4730918407440186, 8.069125175476074, 15.663665771484375, 0.6198129057884216], step: 112200, lr: 5.12839813041253e-05, reference_loss: 29.04166030883789 2023-09-05 01:08:11,870 44k INFO ====> Epoch: 5343, cost 15.09 s 2023-09-05 01:08:26,360 44k INFO ====> Epoch: 5344, cost 14.49 s 2023-09-05 01:08:40,994 44k INFO ====> Epoch: 5345, cost 14.63 s 2023-09-05 01:08:55,425 44k INFO ====> Epoch: 5346, cost 14.43 s 2023-09-05 01:09:09,850 44k INFO ====> Epoch: 5347, cost 14.42 s 2023-09-05 01:09:24,433 44k INFO ====> Epoch: 5348, cost 14.58 s 2023-09-05 01:09:39,018 44k INFO ====> Epoch: 5349, cost 14.58 s 2023-09-05 01:09:53,830 44k INFO ====> Epoch: 5350, cost 14.81 s 2023-09-05 01:10:08,538 44k INFO ====> Epoch: 5351, cost 14.71 s 2023-09-05 01:10:23,106 44k INFO ====> Epoch: 5352, cost 14.57 s 2023-09-05 01:10:28,805 44k INFO Train Epoch: 5353 [33%] 2023-09-05 01:10:28,806 44k INFO Losses: [2.3411293029785156, 2.3836328983306885, 9.39708137512207, 17.13337516784668, 0.596225380897522], step: 112400, lr: 5.121991237452742e-05, reference_loss: 31.851444244384766 2023-09-05 01:10:38,039 44k INFO ====> Epoch: 5353, cost 14.93 s 2023-09-05 01:10:52,823 44k INFO ====> Epoch: 5354, cost 14.78 s 2023-09-05 01:11:07,725 44k INFO ====> Epoch: 5355, cost 14.90 s 2023-09-05 01:11:22,273 44k INFO ====> Epoch: 5356, cost 14.55 s 2023-09-05 01:11:36,718 44k INFO ====> Epoch: 5357, cost 14.44 s 2023-09-05 01:11:51,462 44k INFO ====> Epoch: 5358, cost 14.74 s 2023-09-05 01:12:06,006 44k INFO ====> Epoch: 5359, cost 14.54 s 2023-09-05 01:12:20,607 44k INFO ====> Epoch: 5360, cost 14.60 s 2023-09-05 01:12:35,220 44k INFO ====> Epoch: 5361, cost 14.61 s 2023-09-05 01:12:48,624 44k INFO Train Epoch: 5362 [86%] 2023-09-05 01:12:48,624 44k INFO Losses: [1.956276297569275, 3.135376453399658, 9.15884017944336, 14.807167053222656, 0.6457406282424927], step: 112600, lr: 5.116231877590508e-05, reference_loss: 29.703399658203125 2023-09-05 01:12:50,043 44k INFO ====> Epoch: 5362, cost 14.82 s 2023-09-05 01:13:04,679 44k INFO ====> Epoch: 5363, cost 14.64 s 2023-09-05 01:13:19,199 44k INFO ====> Epoch: 5364, cost 14.52 s 2023-09-05 01:13:33,933 44k INFO ====> Epoch: 5365, cost 14.73 s 2023-09-05 01:13:48,688 44k INFO ====> Epoch: 5366, cost 14.76 s 2023-09-05 01:14:03,303 44k INFO ====> Epoch: 5367, cost 14.61 s 2023-09-05 01:14:18,079 44k INFO ====> Epoch: 5368, cost 14.78 s 2023-09-05 01:14:32,685 44k INFO ====> Epoch: 5369, cost 14.61 s 2023-09-05 01:14:47,307 44k INFO ====> Epoch: 5370, cost 14.62 s 2023-09-05 01:15:02,042 44k INFO ====> Epoch: 5371, cost 14.73 s 2023-09-05 01:15:08,465 44k INFO Train Epoch: 5372 [38%] 2023-09-05 01:15:08,466 44k INFO Losses: [2.12290096282959, 2.742922782897949, 9.128461837768555, 13.8134765625, 0.6008053421974182], step: 112800, lr: 5.109840183895203e-05, reference_loss: 28.408567428588867 2023-09-05 01:15:14,136 44k INFO Saving model and optimizer state at iteration 5372 to ./logs\44k\G_112800.pth 2023-09-05 01:15:14,755 44k INFO Saving model and optimizer state at iteration 5372 to ./logs\44k\D_112800.pth 2023-09-05 01:15:15,981 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_110400.pth 2023-09-05 01:15:16,012 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_110400.pth 2023-09-05 01:15:24,230 44k INFO ====> Epoch: 5372, cost 22.19 s 2023-09-05 01:15:38,769 44k INFO ====> Epoch: 5373, cost 14.54 s 2023-09-05 01:15:53,429 44k INFO ====> Epoch: 5374, cost 14.66 s 2023-09-05 01:16:08,333 44k INFO ====> Epoch: 5375, cost 14.90 s 2023-09-05 01:16:22,852 44k INFO ====> Epoch: 5376, cost 14.52 s 2023-09-05 01:16:37,240 44k INFO ====> Epoch: 5377, cost 14.39 s 2023-09-05 01:16:51,877 44k INFO ====> Epoch: 5378, cost 14.64 s 2023-09-05 01:17:06,399 44k INFO ====> Epoch: 5379, cost 14.52 s 2023-09-05 01:17:21,010 44k INFO ====> Epoch: 5380, cost 14.61 s 2023-09-05 01:17:35,112 44k INFO Train Epoch: 5381 [90%] 2023-09-05 01:17:35,112 44k INFO Losses: [2.257436990737915, 2.644340991973877, 7.677879333496094, 16.1859130859375, 0.764894962310791], step: 113000, lr: 5.1040944871352464e-05, reference_loss: 29.530466079711914 2023-09-05 01:17:35,778 44k INFO ====> Epoch: 5381, cost 14.77 s 2023-09-05 01:17:50,350 44k INFO ====> Epoch: 5382, cost 14.57 s 2023-09-05 01:18:05,033 44k INFO ====> Epoch: 5383, cost 14.68 s 2023-09-05 01:18:19,782 44k INFO ====> Epoch: 5384, cost 14.75 s 2023-09-05 01:18:34,256 44k INFO ====> Epoch: 5385, cost 14.47 s 2023-09-05 01:18:48,970 44k INFO ====> Epoch: 5386, cost 14.71 s 2023-09-05 01:19:03,481 44k INFO ====> Epoch: 5387, cost 14.51 s 2023-09-05 01:19:18,022 44k INFO ====> Epoch: 5388, cost 14.54 s 2023-09-05 01:19:32,538 44k INFO ====> Epoch: 5389, cost 14.52 s 2023-09-05 01:19:47,166 44k INFO ====> Epoch: 5390, cost 14.63 s 2023-09-05 01:19:54,276 44k INFO Train Epoch: 5391 [43%] 2023-09-05 01:19:54,276 44k INFO Losses: [2.48227858543396, 2.3189332485198975, 7.337529182434082, 15.496031761169434, 0.5232898592948914], step: 113200, lr: 5.0977179566467506e-05, reference_loss: 28.158063888549805 2023-09-05 01:20:02,079 44k INFO ====> Epoch: 5391, cost 14.91 s 2023-09-05 01:20:16,705 44k INFO ====> Epoch: 5392, cost 14.63 s 2023-09-05 01:20:31,165 44k INFO ====> Epoch: 5393, cost 14.46 s 2023-09-05 01:20:45,593 44k INFO ====> Epoch: 5394, cost 14.43 s 2023-09-05 01:21:00,040 44k INFO ====> Epoch: 5395, cost 14.45 s 2023-09-05 01:21:14,409 44k INFO ====> Epoch: 5396, cost 14.37 s 2023-09-05 01:21:28,878 44k INFO ====> Epoch: 5397, cost 14.47 s 2023-09-05 01:21:43,201 44k INFO ====> Epoch: 5398, cost 14.32 s 2023-09-05 01:21:57,793 44k INFO ====> Epoch: 5399, cost 14.59 s 2023-09-05 01:22:12,454 44k INFO Train Epoch: 5400 [95%] 2023-09-05 01:22:12,454 44k INFO Losses: [2.134066343307495, 2.644473075866699, 9.229068756103516, 15.09731388092041, 0.48099398612976074], step: 113400, lr: 5.0919858905756846e-05, reference_loss: 29.58591651916504 2023-09-05 01:22:12,715 44k INFO ====> Epoch: 5400, cost 14.92 s 2023-09-05 01:22:27,129 44k INFO ====> Epoch: 5401, cost 14.41 s 2023-09-05 01:22:41,549 44k INFO ====> Epoch: 5402, cost 14.42 s 2023-09-05 01:22:56,228 44k INFO ====> Epoch: 5403, cost 14.68 s 2023-09-05 01:23:10,822 44k INFO ====> Epoch: 5404, cost 14.59 s 2023-09-05 01:23:25,102 44k INFO ====> Epoch: 5405, cost 14.28 s 2023-09-05 01:23:39,531 44k INFO ====> Epoch: 5406, cost 14.43 s 2023-09-05 01:23:54,243 44k INFO ====> Epoch: 5407, cost 14.71 s 2023-09-05 01:24:09,187 44k INFO ====> Epoch: 5408, cost 14.94 s 2023-09-05 01:24:23,694 44k INFO ====> Epoch: 5409, cost 14.51 s 2023-09-05 01:24:31,367 44k INFO Train Epoch: 5410 [48%] 2023-09-05 01:24:31,367 44k INFO Losses: [2.284914970397949, 2.551412343978882, 10.31546401977539, 17.094655990600586, 0.6240844130516052], step: 113600, lr: 5.0856244873218696e-05, reference_loss: 32.87053298950195 2023-09-05 01:24:36,863 44k INFO Saving model and optimizer state at iteration 5410 to ./logs\44k\G_113600.pth 2023-09-05 01:24:37,521 44k INFO Saving model and optimizer state at iteration 5410 to ./logs\44k\D_113600.pth 2023-09-05 01:24:38,644 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_111200.pth 2023-09-05 01:24:38,679 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_111200.pth 2023-09-05 01:24:45,520 44k INFO ====> Epoch: 5410, cost 21.83 s 2023-09-05 01:25:00,252 44k INFO ====> Epoch: 5411, cost 14.73 s 2023-09-05 01:25:14,797 44k INFO ====> Epoch: 5412, cost 14.54 s 2023-09-05 01:25:29,189 44k INFO ====> Epoch: 5413, cost 14.39 s 2023-09-05 01:25:43,823 44k INFO ====> Epoch: 5414, cost 14.63 s 2023-09-05 01:25:58,505 44k INFO ====> Epoch: 5415, cost 14.68 s 2023-09-05 01:26:13,057 44k INFO ====> Epoch: 5416, cost 14.55 s 2023-09-05 01:26:27,341 44k INFO ====> Epoch: 5417, cost 14.28 s 2023-09-05 01:26:41,801 44k INFO ====> Epoch: 5418, cost 14.46 s 2023-09-05 01:26:56,345 44k INFO ====> Epoch: 5419, cost 14.54 s 2023-09-05 01:26:57,029 44k INFO Train Epoch: 5420 [0%] 2023-09-05 01:26:57,029 44k INFO Losses: [2.318845748901367, 2.869607925415039, 8.449010848999023, 14.676212310791016, 0.5559561848640442], step: 113800, lr: 5.0792710313507514e-05, reference_loss: 28.869632720947266 2023-09-05 01:27:11,345 44k INFO ====> Epoch: 5420, cost 15.00 s 2023-09-05 01:27:25,658 44k INFO ====> Epoch: 5421, cost 14.31 s 2023-09-05 01:27:39,991 44k INFO ====> Epoch: 5422, cost 14.33 s 2023-09-05 01:27:54,687 44k INFO ====> Epoch: 5423, cost 14.70 s 2023-09-05 01:28:09,233 44k INFO ====> Epoch: 5424, cost 14.55 s 2023-09-05 01:28:23,651 44k INFO ====> Epoch: 5425, cost 14.42 s 2023-09-05 01:28:38,000 44k INFO ====> Epoch: 5426, cost 14.35 s 2023-09-05 01:28:52,445 44k INFO ====> Epoch: 5427, cost 14.45 s 2023-09-05 01:29:07,129 44k INFO ====> Epoch: 5428, cost 14.68 s 2023-09-05 01:29:15,548 44k INFO Train Epoch: 5429 [52%] 2023-09-05 01:29:15,548 44k INFO Losses: [2.4825098514556885, 2.5210752487182617, 7.823658466339111, 15.939661026000977, 0.6769470572471619], step: 114000, lr: 5.073559707697273e-05, reference_loss: 29.443851470947266 2023-09-05 01:29:21,817 44k INFO ====> Epoch: 5429, cost 14.69 s 2023-09-05 01:29:36,229 44k INFO ====> Epoch: 5430, cost 14.41 s 2023-09-05 01:29:50,748 44k INFO ====> Epoch: 5431, cost 14.52 s 2023-09-05 01:30:05,551 44k INFO ====> Epoch: 5432, cost 14.80 s 2023-09-05 01:30:20,095 44k INFO ====> Epoch: 5433, cost 14.54 s 2023-09-05 01:30:34,657 44k INFO ====> Epoch: 5434, cost 14.56 s 2023-09-05 01:30:49,387 44k INFO ====> Epoch: 5435, cost 14.73 s 2023-09-05 01:31:03,901 44k INFO ====> Epoch: 5436, cost 14.51 s 2023-09-05 01:31:18,288 44k INFO ====> Epoch: 5437, cost 14.39 s 2023-09-05 01:31:32,725 44k INFO ====> Epoch: 5438, cost 14.44 s 2023-09-05 01:31:34,155 44k INFO Train Epoch: 5439 [5%] 2023-09-05 01:31:34,155 44k INFO Losses: [2.3173153400421143, 2.6294236183166504, 8.432003021240234, 16.505687713623047, 0.5547159910202026], step: 114200, lr: 5.0672213242204634e-05, reference_loss: 30.439146041870117 2023-09-05 01:31:47,583 44k INFO ====> Epoch: 5439, cost 14.86 s 2023-09-05 01:32:02,305 44k INFO ====> Epoch: 5440, cost 14.72 s 2023-09-05 01:32:17,002 44k INFO ====> Epoch: 5441, cost 14.70 s 2023-09-05 01:32:31,575 44k INFO ====> Epoch: 5442, cost 14.57 s 2023-09-05 01:32:46,002 44k INFO ====> Epoch: 5443, cost 14.43 s 2023-09-05 01:33:00,521 44k INFO ====> Epoch: 5444, cost 14.52 s 2023-09-05 01:33:14,978 44k INFO ====> Epoch: 5445, cost 14.46 s 2023-09-05 01:33:29,308 44k INFO ====> Epoch: 5446, cost 14.33 s 2023-09-05 01:33:43,704 44k INFO ====> Epoch: 5447, cost 14.40 s 2023-09-05 01:33:52,838 44k INFO Train Epoch: 5448 [57%] 2023-09-05 01:33:52,838 44k INFO Losses: [2.3539981842041016, 2.4128007888793945, 7.286084175109863, 14.266691207885742, 0.754512369632721], step: 114400, lr: 5.0615235497115226e-05, reference_loss: 27.074087142944336 2023-09-05 01:33:58,331 44k INFO Saving model and optimizer state at iteration 5448 to ./logs\44k\G_114400.pth 2023-09-05 01:33:59,027 44k INFO Saving model and optimizer state at iteration 5448 to ./logs\44k\D_114400.pth 2023-09-05 01:34:00,166 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_112000.pth 2023-09-05 01:34:00,200 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_112000.pth 2023-09-05 01:34:05,618 44k INFO ====> Epoch: 5448, cost 21.91 s 2023-09-05 01:34:19,971 44k INFO ====> Epoch: 5449, cost 14.35 s 2023-09-05 01:34:34,338 44k INFO ====> Epoch: 5450, cost 14.37 s 2023-09-05 01:34:48,941 44k INFO ====> Epoch: 5451, cost 14.60 s 2023-09-05 01:35:03,679 44k INFO ====> Epoch: 5452, cost 14.74 s 2023-09-05 01:35:18,203 44k INFO ====> Epoch: 5453, cost 14.52 s 2023-09-05 01:35:32,673 44k INFO ====> Epoch: 5454, cost 14.47 s 2023-09-05 01:35:47,407 44k INFO ====> Epoch: 5455, cost 14.73 s 2023-09-05 01:36:02,137 44k INFO ====> Epoch: 5456, cost 14.73 s 2023-09-05 01:36:16,798 44k INFO ====> Epoch: 5457, cost 14.66 s 2023-09-05 01:36:18,905 44k INFO Train Epoch: 5458 [10%] 2023-09-05 01:36:18,905 44k INFO Losses: [2.3526082038879395, 2.605548143386841, 9.115029335021973, 14.939131736755371, 0.6541147828102112], step: 114600, lr: 5.055200202972092e-05, reference_loss: 29.666433334350586 2023-09-05 01:36:31,488 44k INFO ====> Epoch: 5458, cost 14.69 s 2023-09-05 01:36:46,055 44k INFO ====> Epoch: 5459, cost 14.57 s 2023-09-05 01:37:00,602 44k INFO ====> Epoch: 5460, cost 14.55 s 2023-09-05 01:37:15,170 44k INFO ====> Epoch: 5461, cost 14.57 s 2023-09-05 01:37:29,598 44k INFO ====> Epoch: 5462, cost 14.43 s 2023-09-05 01:37:43,942 44k INFO ====> Epoch: 5463, cost 14.34 s 2023-09-05 01:37:58,701 44k INFO ====> Epoch: 5464, cost 14.76 s 2023-09-05 01:38:13,303 44k INFO ====> Epoch: 5465, cost 14.60 s 2023-09-05 01:38:27,811 44k INFO ====> Epoch: 5466, cost 14.51 s 2023-09-05 01:38:37,718 44k INFO Train Epoch: 5467 [62%] 2023-09-05 01:38:37,719 44k INFO Losses: [2.35646653175354, 2.481287717819214, 9.828363418579102, 17.308269500732422, 0.5507462024688721], step: 114800, lr: 5.049515945464647e-05, reference_loss: 32.5251350402832 2023-09-05 01:38:42,635 44k INFO ====> Epoch: 5467, cost 14.82 s 2023-09-05 01:38:57,026 44k INFO ====> Epoch: 5468, cost 14.39 s 2023-09-05 01:39:11,722 44k INFO ====> Epoch: 5469, cost 14.70 s 2023-09-05 01:39:26,033 44k INFO ====> Epoch: 5470, cost 14.31 s 2023-09-05 01:39:40,413 44k INFO ====> Epoch: 5471, cost 14.38 s 2023-09-05 01:39:54,993 44k INFO ====> Epoch: 5472, cost 14.58 s 2023-09-05 01:40:09,651 44k INFO ====> Epoch: 5473, cost 14.66 s 2023-09-05 01:40:24,185 44k INFO ====> Epoch: 5474, cost 14.53 s 2023-09-05 01:40:38,759 44k INFO ====> Epoch: 5475, cost 14.57 s 2023-09-05 01:40:53,384 44k INFO ====> Epoch: 5476, cost 14.63 s 2023-09-05 01:40:56,209 44k INFO Train Epoch: 5477 [14%] 2023-09-05 01:40:56,210 44k INFO Losses: [2.1341891288757324, 2.775521755218506, 9.762786865234375, 15.356282234191895, 0.5862615704536438], step: 115000, lr: 5.043207599790492e-05, reference_loss: 30.615041732788086 2023-09-05 01:41:08,074 44k INFO ====> Epoch: 5477, cost 14.69 s 2023-09-05 01:41:22,475 44k INFO ====> Epoch: 5478, cost 14.40 s 2023-09-05 01:41:36,780 44k INFO ====> Epoch: 5479, cost 14.30 s 2023-09-05 01:41:51,669 44k INFO ====> Epoch: 5480, cost 14.89 s 2023-09-05 01:42:06,349 44k INFO ====> Epoch: 5481, cost 14.68 s 2023-09-05 01:42:20,853 44k INFO ====> Epoch: 5482, cost 14.50 s 2023-09-05 01:42:35,371 44k INFO ====> Epoch: 5483, cost 14.52 s 2023-09-05 01:42:49,932 44k INFO ====> Epoch: 5484, cost 14.56 s 2023-09-05 01:43:04,501 44k INFO ====> Epoch: 5485, cost 14.57 s 2023-09-05 01:43:15,097 44k INFO Train Epoch: 5486 [67%] 2023-09-05 01:43:15,098 44k INFO Losses: [2.295567512512207, 2.669053316116333, 7.510473251342773, 15.91246223449707, 0.5604104995727539], step: 115200, lr: 5.0375368272177535e-05, reference_loss: 28.947967529296875 2023-09-05 01:43:20,609 44k INFO Saving model and optimizer state at iteration 5486 to ./logs\44k\G_115200.pth 2023-09-05 01:43:21,216 44k INFO Saving model and optimizer state at iteration 5486 to ./logs\44k\D_115200.pth 2023-09-05 01:43:22,454 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_112800.pth 2023-09-05 01:43:22,490 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_112800.pth 2023-09-05 01:43:26,438 44k INFO ====> Epoch: 5486, cost 21.94 s 2023-09-05 01:43:40,870 44k INFO ====> Epoch: 5487, cost 14.43 s 2023-09-05 01:43:55,692 44k INFO ====> Epoch: 5488, cost 14.82 s 2023-09-05 01:44:10,308 44k INFO ====> Epoch: 5489, cost 14.62 s 2023-09-05 01:44:24,706 44k INFO ====> Epoch: 5490, cost 14.40 s 2023-09-05 01:44:39,213 44k INFO ====> Epoch: 5491, cost 14.51 s 2023-09-05 01:44:53,649 44k INFO ====> Epoch: 5492, cost 14.44 s 2023-09-05 01:45:08,303 44k INFO ====> Epoch: 5493, cost 14.65 s 2023-09-05 01:45:22,448 44k INFO ====> Epoch: 5494, cost 14.14 s 2023-09-05 01:45:36,836 44k INFO ====> Epoch: 5495, cost 14.39 s 2023-09-05 01:45:40,342 44k INFO Train Epoch: 5496 [19%] 2023-09-05 01:45:40,343 44k INFO Losses: [2.3839714527130127, 2.3592357635498047, 8.51793098449707, 15.84443473815918, 0.654718279838562], step: 115400, lr: 5.031243447021397e-05, reference_loss: 29.760290145874023 2023-09-05 01:45:51,544 44k INFO ====> Epoch: 5496, cost 14.71 s 2023-09-05 01:46:06,332 44k INFO ====> Epoch: 5497, cost 14.79 s 2023-09-05 01:46:20,731 44k INFO ====> Epoch: 5498, cost 14.40 s 2023-09-05 01:46:35,270 44k INFO ====> Epoch: 5499, cost 14.54 s 2023-09-05 01:46:49,870 44k INFO ====> Epoch: 5500, cost 14.60 s 2023-09-05 01:47:04,598 44k INFO ====> Epoch: 5501, cost 14.73 s 2023-09-05 01:47:19,076 44k INFO ====> Epoch: 5502, cost 14.48 s 2023-09-05 01:47:33,490 44k INFO ====> Epoch: 5503, cost 14.41 s 2023-09-05 01:47:47,943 44k INFO ====> Epoch: 5504, cost 14.45 s 2023-09-05 01:47:59,367 44k INFO Train Epoch: 5505 [71%] 2023-09-05 01:47:59,367 44k INFO Losses: [2.165630578994751, 2.821472406387329, 9.544286727905273, 14.542798042297363, 0.6124194264411926], step: 115600, lr: 5.0255861273926506e-05, reference_loss: 29.686607360839844 2023-09-05 01:48:03,029 44k INFO ====> Epoch: 5505, cost 15.09 s 2023-09-05 01:48:17,772 44k INFO ====> Epoch: 5506, cost 14.74 s 2023-09-05 01:48:32,225 44k INFO ====> Epoch: 5507, cost 14.45 s 2023-09-05 01:48:46,852 44k INFO ====> Epoch: 5508, cost 14.63 s 2023-09-05 01:49:01,584 44k INFO ====> Epoch: 5509, cost 14.73 s 2023-09-05 01:49:16,090 44k INFO ====> Epoch: 5510, cost 14.51 s 2023-09-05 01:49:30,567 44k INFO ====> Epoch: 5511, cost 14.48 s 2023-09-05 01:49:44,971 44k INFO ====> Epoch: 5512, cost 14.40 s 2023-09-05 01:49:59,755 44k INFO ====> Epoch: 5513, cost 14.78 s 2023-09-05 01:50:14,244 44k INFO ====> Epoch: 5514, cost 14.49 s 2023-09-05 01:50:18,520 44k INFO Train Epoch: 5515 [24%] 2023-09-05 01:50:18,520 44k INFO Losses: [2.3518667221069336, 2.559936285018921, 7.758118629455566, 13.255424499511719, 0.5629637241363525], step: 115800, lr: 5.019307677171039e-05, reference_loss: 26.488309860229492 2023-09-05 01:50:29,032 44k INFO ====> Epoch: 5515, cost 14.79 s 2023-09-05 01:50:43,600 44k INFO ====> Epoch: 5516, cost 14.57 s 2023-09-05 01:50:58,215 44k INFO ====> Epoch: 5517, cost 14.62 s 2023-09-05 01:51:12,549 44k INFO ====> Epoch: 5518, cost 14.33 s 2023-09-05 01:51:26,950 44k INFO ====> Epoch: 5519, cost 14.40 s 2023-09-05 01:51:41,348 44k INFO ====> Epoch: 5520, cost 14.40 s 2023-09-05 01:51:56,035 44k INFO ====> Epoch: 5521, cost 14.69 s 2023-09-05 01:52:10,650 44k INFO ====> Epoch: 5522, cost 14.62 s 2023-09-05 01:52:25,088 44k INFO ====> Epoch: 5523, cost 14.44 s 2023-09-05 01:52:37,082 44k INFO Train Epoch: 5524 [76%] 2023-09-05 01:52:37,082 44k INFO Losses: [2.2177274227142334, 2.5589122772216797, 9.835033416748047, 17.67664337158203, 0.615146815776825], step: 116000, lr: 5.013663778571463e-05, reference_loss: 32.90346145629883 2023-09-05 01:52:42,638 44k INFO Saving model and optimizer state at iteration 5524 to ./logs\44k\G_116000.pth 2023-09-05 01:52:43,287 44k INFO Saving model and optimizer state at iteration 5524 to ./logs\44k\D_116000.pth 2023-09-05 01:52:44,994 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_113600.pth 2023-09-05 01:52:45,029 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_113600.pth 2023-09-05 01:52:47,573 44k INFO ====> Epoch: 5524, cost 22.49 s 2023-09-05 01:53:02,062 44k INFO ====> Epoch: 5525, cost 14.49 s 2023-09-05 01:53:16,679 44k INFO ====> Epoch: 5526, cost 14.62 s 2023-09-05 01:53:31,079 44k INFO ====> Epoch: 5527, cost 14.40 s 2023-09-05 01:53:45,637 44k INFO ====> Epoch: 5528, cost 14.56 s 2023-09-05 01:54:00,461 44k INFO ====> Epoch: 5529, cost 14.82 s 2023-09-05 01:54:15,169 44k INFO ====> Epoch: 5530, cost 14.71 s 2023-09-05 01:54:29,583 44k INFO ====> Epoch: 5531, cost 14.41 s 2023-09-05 01:54:44,035 44k INFO ====> Epoch: 5532, cost 14.45 s 2023-09-05 01:54:58,585 44k INFO ====> Epoch: 5533, cost 14.55 s 2023-09-05 01:55:03,438 44k INFO Train Epoch: 5534 [29%] 2023-09-05 01:55:03,439 44k INFO Losses: [2.4594614505767822, 2.3353958129882812, 6.33809232711792, 13.691020965576172, 0.5616873502731323], step: 116200, lr: 5.00740022290577e-05, reference_loss: 25.385658264160156 2023-09-05 01:55:13,307 44k INFO ====> Epoch: 5534, cost 14.72 s 2023-09-05 01:55:27,523 44k INFO ====> Epoch: 5535, cost 14.22 s 2023-09-05 01:55:41,909 44k INFO ====> Epoch: 5536, cost 14.39 s 2023-09-05 01:55:56,559 44k INFO ====> Epoch: 5537, cost 14.65 s 2023-09-05 01:56:11,262 44k INFO ====> Epoch: 5538, cost 14.70 s 2023-09-05 01:56:25,844 44k INFO ====> Epoch: 5539, cost 14.58 s 2023-09-05 01:56:40,259 44k INFO ====> Epoch: 5540, cost 14.41 s 2023-09-05 01:56:54,866 44k INFO ====> Epoch: 5541, cost 14.61 s 2023-09-05 01:57:09,306 44k INFO ====> Epoch: 5542, cost 14.44 s 2023-09-05 01:57:21,805 44k INFO Train Epoch: 5543 [81%] 2023-09-05 01:57:21,805 44k INFO Losses: [2.235966920852661, 2.5021471977233887, 8.782994270324707, 15.010747909545898, 0.5770801305770874], step: 116400, lr: 5.0017697134962515e-05, reference_loss: 29.108936309814453 2023-09-05 01:57:23,891 44k INFO ====> Epoch: 5543, cost 14.58 s 2023-09-05 01:57:38,157 44k INFO ====> Epoch: 5544, cost 14.27 s 2023-09-05 01:57:52,707 44k INFO ====> Epoch: 5545, cost 14.55 s 2023-09-05 01:58:07,251 44k INFO ====> Epoch: 5546, cost 14.54 s 2023-09-05 01:58:21,666 44k INFO ====> Epoch: 5547, cost 14.42 s 2023-09-05 01:58:36,216 44k INFO ====> Epoch: 5548, cost 14.55 s 2023-09-05 01:58:50,689 44k INFO ====> Epoch: 5549, cost 14.47 s 2023-09-05 01:59:05,270 44k INFO ====> Epoch: 5550, cost 14.58 s 2023-09-05 01:59:19,760 44k INFO ====> Epoch: 5551, cost 14.49 s 2023-09-05 01:59:34,218 44k INFO ====> Epoch: 5552, cost 14.46 s 2023-09-05 01:59:39,874 44k INFO Train Epoch: 5553 [33%] 2023-09-05 01:59:39,874 44k INFO Losses: [2.4200549125671387, 2.2861413955688477, 8.69303035736084, 15.902502059936523, 0.5928974151611328], step: 116600, lr: 4.995521017051675e-05, reference_loss: 29.89462661743164 2023-09-05 01:59:49,220 44k INFO ====> Epoch: 5553, cost 15.00 s 2023-09-05 02:00:03,939 44k INFO ====> Epoch: 5554, cost 14.72 s 2023-09-05 02:00:18,675 44k INFO ====> Epoch: 5555, cost 14.74 s 2023-09-05 02:00:33,125 44k INFO ====> Epoch: 5556, cost 14.45 s 2023-09-05 02:00:47,714 44k INFO ====> Epoch: 5557, cost 14.59 s 2023-09-05 02:01:02,283 44k INFO ====> Epoch: 5558, cost 14.57 s 2023-09-05 02:01:16,853 44k INFO ====> Epoch: 5559, cost 14.57 s 2023-09-05 02:01:31,205 44k INFO ====> Epoch: 5560, cost 14.35 s 2023-09-05 02:01:45,521 44k INFO ====> Epoch: 5561, cost 14.32 s 2023-09-05 02:01:59,060 44k INFO Train Epoch: 5562 [86%] 2023-09-05 02:01:59,061 44k INFO Losses: [2.4450719356536865, 2.405027389526367, 9.630790710449219, 15.75784683227539, 0.5868730545043945], step: 116800, lr: 4.989903865068638e-05, reference_loss: 30.825611114501953 2023-09-05 02:02:04,539 44k INFO Saving model and optimizer state at iteration 5562 to ./logs\44k\G_116800.pth 2023-09-05 02:02:05,135 44k INFO Saving model and optimizer state at iteration 5562 to ./logs\44k\D_116800.pth 2023-09-05 02:02:06,829 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_114400.pth 2023-09-05 02:02:06,864 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_114400.pth 2023-09-05 02:02:08,025 44k INFO ====> Epoch: 5562, cost 22.50 s 2023-09-05 02:02:22,507 44k INFO ====> Epoch: 5563, cost 14.48 s 2023-09-05 02:02:36,996 44k INFO ====> Epoch: 5564, cost 14.49 s 2023-09-05 02:02:51,490 44k INFO ====> Epoch: 5565, cost 14.49 s 2023-09-05 02:03:05,800 44k INFO ====> Epoch: 5566, cost 14.31 s 2023-09-05 02:03:20,120 44k INFO ====> Epoch: 5567, cost 14.32 s 2023-09-05 02:03:34,521 44k INFO ====> Epoch: 5568, cost 14.40 s 2023-09-05 02:03:48,987 44k INFO ====> Epoch: 5569, cost 14.47 s 2023-09-05 02:04:03,534 44k INFO ====> Epoch: 5570, cost 14.55 s 2023-09-05 02:04:18,031 44k INFO ====> Epoch: 5571, cost 14.50 s 2023-09-05 02:04:24,457 44k INFO Train Epoch: 5572 [38%] 2023-09-05 02:04:24,457 44k INFO Losses: [2.1633188724517822, 2.9830362796783447, 11.409058570861816, 16.75112533569336, 0.5917065143585205], step: 117000, lr: 4.983669992594203e-05, reference_loss: 33.89824676513672 2023-09-05 02:04:32,981 44k INFO ====> Epoch: 5572, cost 14.95 s 2023-09-05 02:04:47,384 44k INFO ====> Epoch: 5573, cost 14.40 s 2023-09-05 02:05:02,193 44k INFO ====> Epoch: 5574, cost 14.81 s 2023-09-05 02:05:16,788 44k INFO ====> Epoch: 5575, cost 14.59 s 2023-09-05 02:05:31,170 44k INFO ====> Epoch: 5576, cost 14.38 s 2023-09-05 02:05:45,795 44k INFO ====> Epoch: 5577, cost 14.63 s 2023-09-05 02:06:00,470 44k INFO ====> Epoch: 5578, cost 14.67 s 2023-09-05 02:06:14,958 44k INFO ====> Epoch: 5579, cost 14.49 s 2023-09-05 02:06:29,532 44k INFO ====> Epoch: 5580, cost 14.57 s 2023-09-05 02:06:43,647 44k INFO Train Epoch: 5581 [90%] 2023-09-05 02:06:43,647 44k INFO Losses: [2.1909453868865967, 2.751868486404419, 8.420146942138672, 15.538708686828613, 0.5076280236244202], step: 117200, lr: 4.978066166349424e-05, reference_loss: 29.4092960357666 2023-09-05 02:06:44,340 44k INFO ====> Epoch: 5581, cost 14.81 s 2023-09-05 02:06:58,985 44k INFO ====> Epoch: 5582, cost 14.64 s 2023-09-05 02:07:13,651 44k INFO ====> Epoch: 5583, cost 14.67 s 2023-09-05 02:07:27,982 44k INFO ====> Epoch: 5584, cost 14.33 s 2023-09-05 02:07:42,322 44k INFO ====> Epoch: 5585, cost 14.34 s 2023-09-05 02:07:56,855 44k INFO ====> Epoch: 5586, cost 14.53 s 2023-09-05 02:08:11,459 44k INFO ====> Epoch: 5587, cost 14.60 s 2023-09-05 02:08:25,938 44k INFO ====> Epoch: 5588, cost 14.48 s 2023-09-05 02:08:40,337 44k INFO ====> Epoch: 5589, cost 14.40 s 2023-09-05 02:08:54,773 44k INFO ====> Epoch: 5590, cost 14.44 s 2023-09-05 02:09:01,869 44k INFO Train Epoch: 5591 [43%] 2023-09-05 02:09:01,870 44k INFO Losses: [2.092183828353882, 2.8148539066314697, 8.250899314880371, 14.85505485534668, 0.5317736268043518], step: 117400, lr: 4.971847082677779e-05, reference_loss: 28.544767379760742 2023-09-05 02:09:09,783 44k INFO ====> Epoch: 5591, cost 15.01 s 2023-09-05 02:09:24,153 44k INFO ====> Epoch: 5592, cost 14.37 s 2023-09-05 02:09:38,533 44k INFO ====> Epoch: 5593, cost 14.38 s 2023-09-05 02:09:53,205 44k INFO ====> Epoch: 5594, cost 14.67 s 2023-09-05 02:10:07,945 44k INFO ====> Epoch: 5595, cost 14.74 s 2023-09-05 02:10:22,554 44k INFO ====> Epoch: 5596, cost 14.61 s 2023-09-05 02:10:37,316 44k INFO ====> Epoch: 5597, cost 14.76 s 2023-09-05 02:10:51,956 44k INFO ====> Epoch: 5598, cost 14.64 s 2023-09-05 02:11:06,735 44k INFO ====> Epoch: 5599, cost 14.78 s 2023-09-05 02:11:21,341 44k INFO Train Epoch: 5600 [95%] 2023-09-05 02:11:21,341 44k INFO Losses: [2.1435461044311523, 2.6751317977905273, 9.4517240524292, 16.178842544555664, 0.3335094451904297], step: 117600, lr: 4.9662565505582066e-05, reference_loss: 30.78275489807129 2023-09-05 02:11:26,838 44k INFO Saving model and optimizer state at iteration 5600 to ./logs\44k\G_117600.pth 2023-09-05 02:11:27,486 44k INFO Saving model and optimizer state at iteration 5600 to ./logs\44k\D_117600.pth 2023-09-05 02:11:28,693 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_115200.pth 2023-09-05 02:11:28,729 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_115200.pth 2023-09-05 02:11:28,729 44k INFO ====> Epoch: 5600, cost 21.99 s 2023-09-05 02:11:43,230 44k INFO ====> Epoch: 5601, cost 14.50 s 2023-09-05 02:11:58,195 44k INFO ====> Epoch: 5602, cost 14.97 s 2023-09-05 02:12:12,952 44k INFO ====> Epoch: 5603, cost 14.76 s 2023-09-05 02:12:27,399 44k INFO ====> Epoch: 5604, cost 14.45 s 2023-09-05 02:12:42,004 44k INFO ====> Epoch: 5605, cost 14.61 s 2023-09-05 02:12:56,590 44k INFO ====> Epoch: 5606, cost 14.59 s 2023-09-05 02:13:11,358 44k INFO ====> Epoch: 5607, cost 14.77 s 2023-09-05 02:13:25,653 44k INFO ====> Epoch: 5608, cost 14.30 s 2023-09-05 02:13:39,972 44k INFO ====> Epoch: 5609, cost 14.32 s 2023-09-05 02:13:47,819 44k INFO Train Epoch: 5610 [48%] 2023-09-05 02:13:47,819 44k INFO Losses: [2.1953577995300293, 2.6880135536193848, 10.23558521270752, 16.48349380493164, 0.7361131310462952], step: 117800, lr: 4.960052220605432e-05, reference_loss: 32.338565826416016 2023-09-05 02:13:54,999 44k INFO ====> Epoch: 5610, cost 15.03 s 2023-09-05 02:14:09,611 44k INFO ====> Epoch: 5611, cost 14.61 s 2023-09-05 02:14:23,978 44k INFO ====> Epoch: 5612, cost 14.37 s 2023-09-05 02:14:38,291 44k INFO ====> Epoch: 5613, cost 14.31 s 2023-09-05 02:14:53,023 44k INFO ====> Epoch: 5614, cost 14.73 s 2023-09-05 02:15:07,597 44k INFO ====> Epoch: 5615, cost 14.58 s 2023-09-05 02:15:22,008 44k INFO ====> Epoch: 5616, cost 14.41 s 2023-09-05 02:15:36,498 44k INFO ====> Epoch: 5617, cost 14.49 s 2023-09-05 02:15:51,189 44k INFO ====> Epoch: 5618, cost 14.69 s 2023-09-05 02:16:05,726 44k INFO ====> Epoch: 5619, cost 14.54 s 2023-09-05 02:16:06,421 44k INFO Train Epoch: 5620 [0%] 2023-09-05 02:16:06,422 44k INFO Losses: [2.086996078491211, 2.8972461223602295, 8.767260551452637, 14.332365036010742, 0.5721861720085144], step: 118000, lr: 4.9538556417041334e-05, reference_loss: 28.65605354309082 2023-09-05 02:16:20,488 44k INFO ====> Epoch: 5620, cost 14.76 s 2023-09-05 02:16:34,960 44k INFO ====> Epoch: 5621, cost 14.47 s 2023-09-05 02:16:49,519 44k INFO ====> Epoch: 5622, cost 14.56 s 2023-09-05 02:17:04,212 44k INFO ====> Epoch: 5623, cost 14.69 s 2023-09-05 02:17:18,688 44k INFO ====> Epoch: 5624, cost 14.48 s 2023-09-05 02:17:33,245 44k INFO ====> Epoch: 5625, cost 14.56 s 2023-09-05 02:17:47,708 44k INFO ====> Epoch: 5626, cost 14.46 s 2023-09-05 02:18:02,539 44k INFO ====> Epoch: 5627, cost 14.83 s 2023-09-05 02:18:17,213 44k INFO ====> Epoch: 5628, cost 14.67 s 2023-09-05 02:18:25,638 44k INFO Train Epoch: 5629 [52%] 2023-09-05 02:18:25,638 44k INFO Losses: [2.3375918865203857, 2.5981802940368652, 8.367265701293945, 15.44057846069336, 0.4825262725353241], step: 118200, lr: 4.948285339838424e-05, reference_loss: 29.22614288330078 2023-09-05 02:18:31,945 44k INFO ====> Epoch: 5629, cost 14.73 s 2023-09-05 02:18:46,484 44k INFO ====> Epoch: 5630, cost 14.54 s 2023-09-05 02:19:01,151 44k INFO ====> Epoch: 5631, cost 14.67 s 2023-09-05 02:19:15,692 44k INFO ====> Epoch: 5632, cost 14.54 s 2023-09-05 02:19:30,096 44k INFO ====> Epoch: 5633, cost 14.40 s 2023-09-05 02:19:44,494 44k INFO ====> Epoch: 5634, cost 14.40 s 2023-09-05 02:19:59,281 44k INFO ====> Epoch: 5635, cost 14.79 s 2023-09-05 02:20:14,058 44k INFO ====> Epoch: 5636, cost 14.78 s 2023-09-05 02:20:28,486 44k INFO ====> Epoch: 5637, cost 14.43 s 2023-09-05 02:20:42,888 44k INFO ====> Epoch: 5638, cost 14.40 s 2023-09-05 02:20:44,277 44k INFO Train Epoch: 5639 [5%] 2023-09-05 02:20:44,277 44k INFO Losses: [2.2531967163085938, 2.8729934692382812, 8.319198608398438, 15.77748966217041, 0.689824640750885], step: 118400, lr: 4.942103461267253e-05, reference_loss: 29.912702560424805 2023-09-05 02:20:49,864 44k INFO Saving model and optimizer state at iteration 5639 to ./logs\44k\G_118400.pth 2023-09-05 02:20:50,506 44k INFO Saving model and optimizer state at iteration 5639 to ./logs\44k\D_118400.pth 2023-09-05 02:20:51,765 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_116000.pth 2023-09-05 02:20:51,799 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_116000.pth 2023-09-05 02:21:04,949 44k INFO ====> Epoch: 5639, cost 22.06 s 2023-09-05 02:21:19,311 44k INFO ====> Epoch: 5640, cost 14.36 s 2023-09-05 02:21:33,690 44k INFO ====> Epoch: 5641, cost 14.38 s 2023-09-05 02:21:48,356 44k INFO ====> Epoch: 5642, cost 14.67 s 2023-09-05 02:22:02,858 44k INFO ====> Epoch: 5643, cost 14.50 s 2023-09-05 02:22:17,530 44k INFO ====> Epoch: 5644, cost 14.67 s 2023-09-05 02:22:32,092 44k INFO ====> Epoch: 5645, cost 14.56 s 2023-09-05 02:22:46,657 44k INFO ====> Epoch: 5646, cost 14.56 s 2023-09-05 02:23:01,281 44k INFO ====> Epoch: 5647, cost 14.62 s 2023-09-05 02:23:10,726 44k INFO Train Epoch: 5648 [57%] 2023-09-05 02:23:10,726 44k INFO Losses: [2.0606155395507812, 3.0036280155181885, 8.220274925231934, 14.028767585754395, 0.639522910118103], step: 118600, lr: 4.936546373995861e-05, reference_loss: 27.952808380126953 2023-09-05 02:23:16,251 44k INFO ====> Epoch: 5648, cost 14.97 s 2023-09-05 02:23:30,648 44k INFO ====> Epoch: 5649, cost 14.40 s 2023-09-05 02:23:45,189 44k INFO ====> Epoch: 5650, cost 14.54 s 2023-09-05 02:23:59,791 44k INFO ====> Epoch: 5651, cost 14.60 s 2023-09-05 02:24:14,367 44k INFO ====> Epoch: 5652, cost 14.58 s 2023-09-05 02:24:28,927 44k INFO ====> Epoch: 5653, cost 14.56 s 2023-09-05 02:24:43,603 44k INFO ====> Epoch: 5654, cost 14.68 s 2023-09-05 02:24:58,297 44k INFO ====> Epoch: 5655, cost 14.69 s 2023-09-05 02:25:12,750 44k INFO ====> Epoch: 5656, cost 14.45 s 2023-09-05 02:25:27,182 44k INFO ====> Epoch: 5657, cost 14.43 s 2023-09-05 02:25:29,287 44k INFO Train Epoch: 5658 [10%] 2023-09-05 02:25:29,287 44k INFO Losses: [2.1919004917144775, 2.7034835815429688, 10.487701416015625, 15.867892265319824, 0.5947934985160828], step: 118800, lr: 4.930379160880784e-05, reference_loss: 31.84576988220215 2023-09-05 02:25:41,795 44k INFO ====> Epoch: 5658, cost 14.61 s 2023-09-05 02:25:56,408 44k INFO ====> Epoch: 5659, cost 14.61 s 2023-09-05 02:26:11,071 44k INFO ====> Epoch: 5660, cost 14.66 s 2023-09-05 02:26:25,537 44k INFO ====> Epoch: 5661, cost 14.47 s 2023-09-05 02:26:40,086 44k INFO ====> Epoch: 5662, cost 14.55 s 2023-09-05 02:26:54,547 44k INFO ====> Epoch: 5663, cost 14.46 s 2023-09-05 02:27:09,187 44k INFO ====> Epoch: 5664, cost 14.64 s 2023-09-05 02:27:23,622 44k INFO ====> Epoch: 5665, cost 14.44 s 2023-09-05 02:27:38,032 44k INFO ====> Epoch: 5666, cost 14.41 s 2023-09-05 02:27:47,792 44k INFO Train Epoch: 5667 [62%] 2023-09-05 02:27:47,792 44k INFO Losses: [2.2374515533447266, 2.606968641281128, 9.171860694885254, 14.728350639343262, 0.524569571018219], step: 119000, lr: 4.924835256854331e-05, reference_loss: 29.269203186035156 2023-09-05 02:27:52,685 44k INFO ====> Epoch: 5667, cost 14.65 s 2023-09-05 02:28:07,236 44k INFO ====> Epoch: 5668, cost 14.55 s 2023-09-05 02:28:21,647 44k INFO ====> Epoch: 5669, cost 14.41 s 2023-09-05 02:28:36,114 44k INFO ====> Epoch: 5670, cost 14.47 s 2023-09-05 02:28:50,793 44k INFO ====> Epoch: 5671, cost 14.68 s 2023-09-05 02:29:05,426 44k INFO ====> Epoch: 5672, cost 14.63 s 2023-09-05 02:29:19,914 44k INFO ====> Epoch: 5673, cost 14.49 s 2023-09-05 02:29:34,335 44k INFO ====> Epoch: 5674, cost 14.42 s 2023-09-05 02:29:49,026 44k INFO ====> Epoch: 5675, cost 14.69 s 2023-09-05 02:30:03,986 44k INFO ====> Epoch: 5676, cost 14.96 s 2023-09-05 02:30:06,946 44k INFO Train Epoch: 5677 [14%] 2023-09-05 02:30:06,946 44k INFO Losses: [2.3866162300109863, 2.314539909362793, 8.217761039733887, 15.487061500549316, 0.6737229228019714], step: 119200, lr: 4.918682674404045e-05, reference_loss: 29.079702377319336 2023-09-05 02:30:12,364 44k INFO Saving model and optimizer state at iteration 5677 to ./logs\44k\G_119200.pth 2023-09-05 02:30:12,996 44k INFO Saving model and optimizer state at iteration 5677 to ./logs\44k\D_119200.pth 2023-09-05 02:30:14,199 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_116800.pth 2023-09-05 02:30:14,235 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_116800.pth 2023-09-05 02:30:25,904 44k INFO ====> Epoch: 5677, cost 21.92 s 2023-09-05 02:30:40,567 44k INFO ====> Epoch: 5678, cost 14.66 s 2023-09-05 02:30:55,066 44k INFO ====> Epoch: 5679, cost 14.50 s 2023-09-05 02:31:09,628 44k INFO ====> Epoch: 5680, cost 14.56 s 2023-09-05 02:31:24,074 44k INFO ====> Epoch: 5681, cost 14.45 s 2023-09-05 02:31:38,323 44k INFO ====> Epoch: 5682, cost 14.25 s 2023-09-05 02:31:53,033 44k INFO ====> Epoch: 5683, cost 14.71 s 2023-09-05 02:32:07,585 44k INFO ====> Epoch: 5684, cost 14.55 s 2023-09-05 02:32:22,091 44k INFO ====> Epoch: 5685, cost 14.51 s 2023-09-05 02:32:32,642 44k INFO Train Epoch: 5686 [67%] 2023-09-05 02:32:32,643 44k INFO Losses: [1.9465545415878296, 2.89106822013855, 9.327049255371094, 15.234370231628418, 0.5538470149040222], step: 119400, lr: 4.913151922347523e-05, reference_loss: 29.95288848876953 2023-09-05 02:32:36,831 44k INFO ====> Epoch: 5686, cost 14.74 s 2023-09-05 02:32:51,362 44k INFO ====> Epoch: 5687, cost 14.53 s 2023-09-05 02:33:05,676 44k INFO ====> Epoch: 5688, cost 14.31 s 2023-09-05 02:33:19,980 44k INFO ====> Epoch: 5689, cost 14.30 s 2023-09-05 02:33:34,426 44k INFO ====> Epoch: 5690, cost 14.45 s 2023-09-05 02:33:48,887 44k INFO ====> Epoch: 5691, cost 14.46 s 2023-09-05 02:34:03,425 44k INFO ====> Epoch: 5692, cost 14.54 s 2023-09-05 02:34:18,108 44k INFO ====> Epoch: 5693, cost 14.68 s 2023-09-05 02:34:32,577 44k INFO ====> Epoch: 5694, cost 14.47 s 2023-09-05 02:34:47,208 44k INFO ====> Epoch: 5695, cost 14.63 s 2023-09-05 02:34:50,709 44k INFO Train Epoch: 5696 [19%] 2023-09-05 02:34:50,710 44k INFO Losses: [2.3815383911132812, 2.401885509490967, 8.715678215026855, 16.555404663085938, 0.5633392930030823], step: 119600, lr: 4.907013935853264e-05, reference_loss: 30.61784553527832 2023-09-05 02:35:02,157 44k INFO ====> Epoch: 5696, cost 14.95 s 2023-09-05 02:35:16,801 44k INFO ====> Epoch: 5697, cost 14.64 s 2023-09-05 02:35:31,173 44k INFO ====> Epoch: 5698, cost 14.37 s 2023-09-05 02:35:45,743 44k INFO ====> Epoch: 5699, cost 14.57 s 2023-09-05 02:36:00,520 44k INFO ====> Epoch: 5700, cost 14.78 s 2023-09-05 02:36:15,092 44k INFO ====> Epoch: 5701, cost 14.57 s 2023-09-05 02:36:29,657 44k INFO ====> Epoch: 5702, cost 14.56 s 2023-09-05 02:36:44,174 44k INFO ====> Epoch: 5703, cost 14.52 s 2023-09-05 02:36:58,982 44k INFO ====> Epoch: 5704, cost 14.81 s 2023-09-05 02:37:10,433 44k INFO Train Epoch: 5705 [71%] 2023-09-05 02:37:10,433 44k INFO Losses: [1.898937702178955, 3.2590413093566895, 10.608827590942383, 14.893208503723145, 0.5100274085998535], step: 119800, lr: 4.9014963045658596e-05, reference_loss: 31.170042037963867 2023-09-05 02:37:13,884 44k INFO ====> Epoch: 5705, cost 14.90 s 2023-09-05 02:37:28,398 44k INFO ====> Epoch: 5706, cost 14.52 s 2023-09-05 02:37:42,845 44k INFO ====> Epoch: 5707, cost 14.45 s 2023-09-05 02:37:57,471 44k INFO ====> Epoch: 5708, cost 14.63 s 2023-09-05 02:38:12,042 44k INFO ====> Epoch: 5709, cost 14.57 s 2023-09-05 02:38:26,488 44k INFO ====> Epoch: 5710, cost 14.45 s 2023-09-05 02:38:40,940 44k INFO ====> Epoch: 5711, cost 14.45 s 2023-09-05 02:38:55,526 44k INFO ====> Epoch: 5712, cost 14.59 s 2023-09-05 02:39:10,028 44k INFO ====> Epoch: 5713, cost 14.50 s 2023-09-05 02:39:24,244 44k INFO ====> Epoch: 5714, cost 14.22 s 2023-09-05 02:39:28,426 44k INFO Train Epoch: 5715 [24%] 2023-09-05 02:39:28,427 44k INFO Losses: [2.2639999389648438, 2.6403205394744873, 7.542764186859131, 15.65700626373291, 0.6994075179100037], step: 120000, lr: 4.895372879401202e-05, reference_loss: 28.803499221801758 2023-09-05 02:39:33,816 44k INFO Saving model and optimizer state at iteration 5715 to ./logs\44k\G_120000.pth 2023-09-05 02:39:34,513 44k INFO Saving model and optimizer state at iteration 5715 to ./logs\44k\D_120000.pth 2023-09-05 02:39:35,765 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_117600.pth 2023-09-05 02:39:35,806 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_117600.pth 2023-09-05 02:39:46,160 44k INFO ====> Epoch: 5715, cost 21.92 s 2023-09-05 02:40:00,711 44k INFO ====> Epoch: 5716, cost 14.55 s 2023-09-05 02:40:15,234 44k INFO ====> Epoch: 5717, cost 14.52 s 2023-09-05 02:40:29,771 44k INFO ====> Epoch: 5718, cost 14.54 s 2023-09-05 02:40:44,290 44k INFO ====> Epoch: 5719, cost 14.52 s 2023-09-05 02:40:59,046 44k INFO ====> Epoch: 5720, cost 14.76 s 2023-09-05 02:41:13,476 44k INFO ====> Epoch: 5721, cost 14.43 s 2023-09-05 02:41:27,781 44k INFO ====> Epoch: 5722, cost 14.31 s 2023-09-05 02:41:42,104 44k INFO ====> Epoch: 5723, cost 14.32 s 2023-09-05 02:41:54,416 44k INFO Train Epoch: 5724 [76%] 2023-09-05 02:41:54,417 44k INFO Losses: [2.334127426147461, 2.458573818206787, 8.290939331054688, 15.987032890319824, 0.7303446531295776], step: 120200, lr: 4.8898683377561216e-05, reference_loss: 29.8010196685791 2023-09-05 02:41:57,240 44k INFO ====> Epoch: 5724, cost 15.14 s 2023-09-05 02:42:12,135 44k INFO ====> Epoch: 5725, cost 14.89 s 2023-09-05 02:42:26,660 44k INFO ====> Epoch: 5726, cost 14.53 s 2023-09-05 02:42:41,024 44k INFO ====> Epoch: 5727, cost 14.36 s 2023-09-05 02:42:55,664 44k INFO ====> Epoch: 5728, cost 14.64 s 2023-09-05 02:43:10,322 44k INFO ====> Epoch: 5729, cost 14.66 s 2023-09-05 02:43:24,758 44k INFO ====> Epoch: 5730, cost 14.44 s 2023-09-05 02:43:39,286 44k INFO ====> Epoch: 5731, cost 14.53 s 2023-09-05 02:43:54,042 44k INFO ====> Epoch: 5732, cost 14.76 s 2023-09-05 02:44:08,679 44k INFO ====> Epoch: 5733, cost 14.64 s 2023-09-05 02:44:13,750 44k INFO Train Epoch: 5734 [29%] 2023-09-05 02:44:13,750 44k INFO Losses: [2.3241381645202637, 2.375206470489502, 8.389935493469238, 15.571667671203613, 0.5694168210029602], step: 120400, lr: 4.8837594393767875e-05, reference_loss: 29.230363845825195 2023-09-05 02:44:23,561 44k INFO ====> Epoch: 5734, cost 14.88 s 2023-09-05 02:44:38,089 44k INFO ====> Epoch: 5735, cost 14.53 s 2023-09-05 02:44:52,528 44k INFO ====> Epoch: 5736, cost 14.44 s 2023-09-05 02:45:07,066 44k INFO ====> Epoch: 5737, cost 14.54 s 2023-09-05 02:45:21,436 44k INFO ====> Epoch: 5738, cost 14.37 s 2023-09-05 02:45:35,814 44k INFO ====> Epoch: 5739, cost 14.38 s 2023-09-05 02:45:50,497 44k INFO ====> Epoch: 5740, cost 14.68 s 2023-09-05 02:46:05,026 44k INFO ====> Epoch: 5741, cost 14.53 s 2023-09-05 02:46:19,458 44k INFO ====> Epoch: 5742, cost 14.43 s 2023-09-05 02:46:32,046 44k INFO Train Epoch: 5743 [81%] 2023-09-05 02:46:32,046 44k INFO Losses: [2.3386051654815674, 2.2872390747070312, 7.555541515350342, 14.55374526977539, 0.6925609111785889], step: 120600, lr: 4.878267956321079e-05, reference_loss: 27.427690505981445 2023-09-05 02:46:34,271 44k INFO ====> Epoch: 5743, cost 14.81 s 2023-09-05 02:46:48,802 44k INFO ====> Epoch: 5744, cost 14.53 s 2023-09-05 02:47:03,449 44k INFO ====> Epoch: 5745, cost 14.65 s 2023-09-05 02:47:18,066 44k INFO ====> Epoch: 5746, cost 14.62 s 2023-09-05 02:47:32,401 44k INFO ====> Epoch: 5747, cost 14.34 s 2023-09-05 02:47:47,107 44k INFO ====> Epoch: 5748, cost 14.71 s 2023-09-05 02:48:01,982 44k INFO ====> Epoch: 5749, cost 14.87 s 2023-09-05 02:48:16,555 44k INFO ====> Epoch: 5750, cost 14.57 s 2023-09-05 02:48:31,072 44k INFO ====> Epoch: 5751, cost 14.52 s 2023-09-05 02:48:45,743 44k INFO ====> Epoch: 5752, cost 14.67 s 2023-09-05 02:48:51,390 44k INFO Train Epoch: 5753 [33%] 2023-09-05 02:48:51,391 44k INFO Losses: [2.2705743312835693, 2.6857986450195312, 9.480371475219727, 16.238962173461914, 0.6633535027503967], step: 120800, lr: 4.872173550264739e-05, reference_loss: 31.339059829711914 2023-09-05 02:48:56,822 44k INFO Saving model and optimizer state at iteration 5753 to ./logs\44k\G_120800.pth 2023-09-05 02:48:57,485 44k INFO Saving model and optimizer state at iteration 5753 to ./logs\44k\D_120800.pth 2023-09-05 02:48:58,774 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_118400.pth 2023-09-05 02:48:58,810 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_118400.pth 2023-09-05 02:49:07,906 44k INFO ====> Epoch: 5753, cost 22.16 s 2023-09-05 02:49:22,316 44k INFO ====> Epoch: 5754, cost 14.41 s 2023-09-05 02:49:36,750 44k INFO ====> Epoch: 5755, cost 14.43 s 2023-09-05 02:49:51,319 44k INFO ====> Epoch: 5756, cost 14.57 s 2023-09-05 02:50:06,118 44k INFO ====> Epoch: 5757, cost 14.80 s 2023-09-05 02:50:20,556 44k INFO ====> Epoch: 5758, cost 14.44 s 2023-09-05 02:50:35,109 44k INFO ====> Epoch: 5759, cost 14.55 s 2023-09-05 02:50:49,399 44k INFO ====> Epoch: 5760, cost 14.29 s 2023-09-05 02:51:04,008 44k INFO ====> Epoch: 5761, cost 14.61 s 2023-09-05 02:51:17,379 44k INFO Train Epoch: 5762 [86%] 2023-09-05 02:51:17,379 44k INFO Losses: [1.8954746723175049, 3.1493983268737793, 11.509225845336914, 15.893773078918457, 0.7645911574363708], step: 121000, lr: 4.86669509481912e-05, reference_loss: 33.21246337890625 2023-09-05 02:51:18,715 44k INFO ====> Epoch: 5762, cost 14.71 s 2023-09-05 02:51:33,076 44k INFO ====> Epoch: 5763, cost 14.36 s 2023-09-05 02:51:47,383 44k INFO ====> Epoch: 5764, cost 14.31 s 2023-09-05 02:52:02,014 44k INFO ====> Epoch: 5765, cost 14.63 s 2023-09-05 02:52:16,527 44k INFO ====> Epoch: 5766, cost 14.51 s 2023-09-05 02:52:30,948 44k INFO ====> Epoch: 5767, cost 14.42 s 2023-09-05 02:52:45,485 44k INFO ====> Epoch: 5768, cost 14.54 s 2023-09-05 02:53:00,110 44k INFO ====> Epoch: 5769, cost 14.63 s 2023-09-05 02:53:14,775 44k INFO ====> Epoch: 5770, cost 14.66 s 2023-09-05 02:53:29,169 44k INFO ====> Epoch: 5771, cost 14.39 s 2023-09-05 02:53:35,387 44k INFO Train Epoch: 5772 [38%] 2023-09-05 02:53:35,387 44k INFO Losses: [2.1978564262390137, 2.5587778091430664, 9.806389808654785, 15.94006633758545, 0.5611137747764587], step: 121200, lr: 4.8606151467052004e-05, reference_loss: 31.0642032623291 2023-09-05 02:53:43,944 44k INFO ====> Epoch: 5772, cost 14.78 s 2023-09-05 02:53:58,685 44k INFO ====> Epoch: 5773, cost 14.74 s 2023-09-05 02:54:13,428 44k INFO ====> Epoch: 5774, cost 14.74 s 2023-09-05 02:54:28,045 44k INFO ====> Epoch: 5775, cost 14.62 s 2023-09-05 02:54:42,538 44k INFO ====> Epoch: 5776, cost 14.49 s 2023-09-05 02:54:57,156 44k INFO ====> Epoch: 5777, cost 14.62 s 2023-09-05 02:55:11,769 44k INFO ====> Epoch: 5778, cost 14.61 s 2023-09-05 02:55:26,066 44k INFO ====> Epoch: 5779, cost 14.30 s 2023-09-05 02:55:40,416 44k INFO ====> Epoch: 5780, cost 14.35 s 2023-09-05 02:55:54,898 44k INFO Train Epoch: 5781 [90%] 2023-09-05 02:55:54,899 44k INFO Losses: [2.3218302726745605, 2.632779359817505, 9.899925231933594, 16.479646682739258, 0.4206290543079376], step: 121400, lr: 4.85514968796388e-05, reference_loss: 31.754812240600586 2023-09-05 02:55:55,623 44k INFO ====> Epoch: 5781, cost 15.21 s 2023-09-05 02:56:10,362 44k INFO ====> Epoch: 5782, cost 14.74 s 2023-09-05 02:56:24,913 44k INFO ====> Epoch: 5783, cost 14.55 s 2023-09-05 02:56:39,383 44k INFO ====> Epoch: 5784, cost 14.47 s 2023-09-05 02:56:53,863 44k INFO ====> Epoch: 5785, cost 14.48 s 2023-09-05 02:57:08,417 44k INFO ====> Epoch: 5786, cost 14.55 s 2023-09-05 02:57:22,782 44k INFO ====> Epoch: 5787, cost 14.36 s 2023-09-05 02:57:37,313 44k INFO ====> Epoch: 5788, cost 14.53 s 2023-09-05 02:57:51,808 44k INFO ====> Epoch: 5789, cost 14.50 s 2023-09-05 02:58:06,577 44k INFO ====> Epoch: 5790, cost 14.77 s 2023-09-05 02:58:13,566 44k INFO Train Epoch: 5791 [43%] 2023-09-05 02:58:13,566 44k INFO Losses: [2.130998134613037, 3.054471969604492, 10.013473510742188, 14.653519630432129, 0.764569103717804], step: 121600, lr: 4.84908416349337e-05, reference_loss: 30.61703109741211 2023-09-05 02:58:19,081 44k INFO Saving model and optimizer state at iteration 5791 to ./logs\44k\G_121600.pth 2023-09-05 02:58:19,707 44k INFO Saving model and optimizer state at iteration 5791 to ./logs\44k\D_121600.pth 2023-09-05 02:58:20,862 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_119200.pth 2023-09-05 02:58:20,900 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_119200.pth 2023-09-05 02:58:28,373 44k INFO ====> Epoch: 5791, cost 21.80 s 2023-09-05 02:58:42,952 44k INFO ====> Epoch: 5792, cost 14.58 s 2023-09-05 02:58:57,740 44k INFO ====> Epoch: 5793, cost 14.79 s 2023-09-05 02:59:12,399 44k INFO ====> Epoch: 5794, cost 14.66 s 2023-09-05 02:59:26,624 44k INFO ====> Epoch: 5795, cost 14.22 s 2023-09-05 02:59:40,985 44k INFO ====> Epoch: 5796, cost 14.36 s 2023-09-05 02:59:55,825 44k INFO ====> Epoch: 5797, cost 14.84 s 2023-09-05 03:00:10,705 44k INFO ====> Epoch: 5798, cost 14.88 s 2023-09-05 03:00:25,171 44k INFO ====> Epoch: 5799, cost 14.47 s 2023-09-05 03:00:39,717 44k INFO Train Epoch: 5800 [95%] 2023-09-05 03:00:39,718 44k INFO Losses: [2.1889288425445557, 2.6511330604553223, 10.3828125, 15.527237892150879, 0.5818085670471191], step: 121800, lr: 4.843631670623876e-05, reference_loss: 31.331920623779297 2023-09-05 03:00:39,981 44k INFO ====> Epoch: 5800, cost 14.81 s 2023-09-05 03:00:54,649 44k INFO ====> Epoch: 5801, cost 14.67 s 2023-09-05 03:01:09,323 44k INFO ====> Epoch: 5802, cost 14.67 s 2023-09-05 03:01:23,769 44k INFO ====> Epoch: 5803, cost 14.45 s 2023-09-05 03:01:38,185 44k INFO ====> Epoch: 5804, cost 14.42 s 2023-09-05 03:01:52,804 44k INFO ====> Epoch: 5805, cost 14.62 s 2023-09-05 03:02:07,557 44k INFO ====> Epoch: 5806, cost 14.75 s 2023-09-05 03:02:22,026 44k INFO ====> Epoch: 5807, cost 14.47 s 2023-09-05 03:02:36,403 44k INFO ====> Epoch: 5808, cost 14.38 s 2023-09-05 03:02:51,160 44k INFO ====> Epoch: 5809, cost 14.76 s 2023-09-05 03:02:58,920 44k INFO Train Epoch: 5810 [48%] 2023-09-05 03:02:58,921 44k INFO Losses: [2.3626861572265625, 2.5519039630889893, 10.134337425231934, 16.85107421875, 0.6184474229812622], step: 122000, lr: 4.837580535579135e-05, reference_loss: 32.51844787597656 2023-09-05 03:03:06,022 44k INFO ====> Epoch: 5810, cost 14.86 s 2023-09-05 03:03:20,598 44k INFO ====> Epoch: 5811, cost 14.58 s 2023-09-05 03:03:35,196 44k INFO ====> Epoch: 5812, cost 14.60 s 2023-09-05 03:03:49,677 44k INFO ====> Epoch: 5813, cost 14.48 s 2023-09-05 03:04:04,337 44k INFO ====> Epoch: 5814, cost 14.66 s 2023-09-05 03:04:18,956 44k INFO ====> Epoch: 5815, cost 14.62 s 2023-09-05 03:04:33,359 44k INFO ====> Epoch: 5816, cost 14.40 s 2023-09-05 03:04:47,925 44k INFO ====> Epoch: 5817, cost 14.57 s 2023-09-05 03:05:02,563 44k INFO ====> Epoch: 5818, cost 14.64 s 2023-09-05 03:05:17,067 44k INFO ====> Epoch: 5819, cost 14.50 s 2023-09-05 03:05:17,745 44k INFO Train Epoch: 5820 [0%] 2023-09-05 03:05:17,745 44k INFO Losses: [1.742004156112671, 3.3900246620178223, 11.009283065795898, 15.279806137084961, 0.6921348571777344], step: 122200, lr: 4.831536960199913e-05, reference_loss: 32.113250732421875 2023-09-05 03:05:31,798 44k INFO ====> Epoch: 5820, cost 14.73 s 2023-09-05 03:05:46,311 44k INFO ====> Epoch: 5821, cost 14.51 s 2023-09-05 03:06:01,122 44k INFO ====> Epoch: 5822, cost 14.81 s 2023-09-05 03:06:15,795 44k INFO ====> Epoch: 5823, cost 14.67 s 2023-09-05 03:06:30,353 44k INFO ====> Epoch: 5824, cost 14.56 s 2023-09-05 03:06:44,900 44k INFO ====> Epoch: 5825, cost 14.55 s 2023-09-05 03:06:59,474 44k INFO ====> Epoch: 5826, cost 14.57 s 2023-09-05 03:07:14,109 44k INFO ====> Epoch: 5827, cost 14.64 s 2023-09-05 03:07:28,429 44k INFO ====> Epoch: 5828, cost 14.32 s 2023-09-05 03:07:36,793 44k INFO Train Epoch: 5829 [52%] 2023-09-05 03:07:36,794 44k INFO Losses: [2.043318033218384, 3.0825276374816895, 11.32083511352539, 16.42715835571289, 0.6590529680252075], step: 122400, lr: 4.826104198066701e-05, reference_loss: 33.532894134521484 2023-09-05 03:07:42,243 44k INFO Saving model and optimizer state at iteration 5829 to ./logs\44k\G_122400.pth 2023-09-05 03:07:42,808 44k INFO Saving model and optimizer state at iteration 5829 to ./logs\44k\D_122400.pth 2023-09-05 03:07:44,183 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_120000.pth 2023-09-05 03:07:44,222 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_120000.pth 2023-09-05 03:07:50,374 44k INFO ====> Epoch: 5829, cost 21.95 s 2023-09-05 03:08:05,141 44k INFO ====> Epoch: 5830, cost 14.77 s 2023-09-05 03:08:20,146 44k INFO ====> Epoch: 5831, cost 15.01 s 2023-09-05 03:08:34,916 44k INFO ====> Epoch: 5832, cost 14.77 s 2023-09-05 03:08:49,429 44k INFO ====> Epoch: 5833, cost 14.51 s 2023-09-05 03:09:03,987 44k INFO ====> Epoch: 5834, cost 14.56 s 2023-09-05 03:09:18,700 44k INFO ====> Epoch: 5835, cost 14.71 s 2023-09-05 03:09:33,174 44k INFO ====> Epoch: 5836, cost 14.47 s 2023-09-05 03:09:47,475 44k INFO ====> Epoch: 5837, cost 14.30 s 2023-09-05 03:10:02,012 44k INFO ====> Epoch: 5838, cost 14.54 s 2023-09-05 03:10:03,413 44k INFO Train Epoch: 5839 [5%] 2023-09-05 03:10:03,414 44k INFO Losses: [2.1626360416412354, 2.977166175842285, 8.192718505859375, 13.915994644165039, 0.6568475365638733], step: 122600, lr: 4.820074960042759e-05, reference_loss: 27.905363082885742 2023-09-05 03:10:16,729 44k INFO ====> Epoch: 5839, cost 14.72 s 2023-09-05 03:10:30,975 44k INFO ====> Epoch: 5840, cost 14.25 s 2023-09-05 03:10:45,594 44k INFO ====> Epoch: 5841, cost 14.62 s 2023-09-05 03:11:00,288 44k INFO ====> Epoch: 5842, cost 14.69 s 2023-09-05 03:11:14,942 44k INFO ====> Epoch: 5843, cost 14.65 s 2023-09-05 03:11:29,239 44k INFO ====> Epoch: 5844, cost 14.30 s 2023-09-05 03:11:43,797 44k INFO ====> Epoch: 5845, cost 14.56 s 2023-09-05 03:11:58,476 44k INFO ====> Epoch: 5846, cost 14.68 s 2023-09-05 03:12:13,051 44k INFO ====> Epoch: 5847, cost 14.57 s 2023-09-05 03:12:22,262 44k INFO Train Epoch: 5848 [57%] 2023-09-05 03:12:22,262 44k INFO Losses: [2.450927257537842, 2.624178886413574, 8.655245780944824, 15.939309120178223, 0.5407566428184509], step: 122800, lr: 4.814655086214229e-05, reference_loss: 30.210418701171875 2023-09-05 03:12:27,905 44k INFO ====> Epoch: 5848, cost 14.85 s 2023-09-05 03:12:42,555 44k INFO ====> Epoch: 5849, cost 14.65 s 2023-09-05 03:12:57,238 44k INFO ====> Epoch: 5850, cost 14.68 s 2023-09-05 03:13:11,973 44k INFO ====> Epoch: 5851, cost 14.73 s 2023-09-05 03:13:26,310 44k INFO ====> Epoch: 5852, cost 14.34 s 2023-09-05 03:13:40,790 44k INFO ====> Epoch: 5853, cost 14.48 s 2023-09-05 03:13:55,565 44k INFO ====> Epoch: 5854, cost 14.77 s 2023-09-05 03:14:10,267 44k INFO ====> Epoch: 5855, cost 14.70 s 2023-09-05 03:14:24,718 44k INFO ====> Epoch: 5856, cost 14.45 s 2023-09-05 03:14:39,122 44k INFO ====> Epoch: 5857, cost 14.40 s 2023-09-05 03:14:41,220 44k INFO Train Epoch: 5858 [10%] 2023-09-05 03:14:41,220 44k INFO Losses: [2.3363826274871826, 2.6825027465820312, 8.075526237487793, 14.553915977478027, 0.5983315706253052], step: 123000, lr: 4.808640151532628e-05, reference_loss: 28.246658325195312 2023-09-05 03:14:53,913 44k INFO ====> Epoch: 5858, cost 14.79 s 2023-09-05 03:15:08,600 44k INFO ====> Epoch: 5859, cost 14.69 s 2023-09-05 03:15:22,846 44k INFO ====> Epoch: 5860, cost 14.25 s 2023-09-05 03:15:37,032 44k INFO ====> Epoch: 5861, cost 14.19 s 2023-09-05 03:15:51,777 44k INFO ====> Epoch: 5862, cost 14.75 s 2023-09-05 03:16:06,450 44k INFO ====> Epoch: 5863, cost 14.67 s 2023-09-05 03:16:20,846 44k INFO ====> Epoch: 5864, cost 14.40 s 2023-09-05 03:16:35,295 44k INFO ====> Epoch: 5865, cost 14.45 s 2023-09-05 03:16:49,842 44k INFO ====> Epoch: 5866, cost 14.55 s 2023-09-05 03:16:59,714 44k INFO Train Epoch: 5867 [62%] 2023-09-05 03:16:59,714 44k INFO Losses: [2.3838307857513428, 2.3296289443969727, 7.8866496086120605, 15.191513061523438, 0.615468442440033], step: 123200, lr: 4.8032331354334674e-05, reference_loss: 28.40709114074707 2023-09-05 03:17:05,296 44k INFO Saving model and optimizer state at iteration 5867 to ./logs\44k\G_123200.pth 2023-09-05 03:17:05,947 44k INFO Saving model and optimizer state at iteration 5867 to ./logs\44k\D_123200.pth 2023-09-05 03:17:07,074 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_120800.pth 2023-09-05 03:17:07,108 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_120800.pth 2023-09-05 03:17:11,802 44k INFO ====> Epoch: 5867, cost 21.96 s 2023-09-05 03:17:26,225 44k INFO ====> Epoch: 5868, cost 14.42 s 2023-09-05 03:17:40,844 44k INFO ====> Epoch: 5869, cost 14.62 s 2023-09-05 03:17:55,614 44k INFO ====> Epoch: 5870, cost 14.77 s 2023-09-05 03:18:10,278 44k INFO ====> Epoch: 5871, cost 14.66 s 2023-09-05 03:18:24,780 44k INFO ====> Epoch: 5872, cost 14.50 s 2023-09-05 03:18:39,480 44k INFO ====> Epoch: 5873, cost 14.70 s 2023-09-05 03:18:54,034 44k INFO ====> Epoch: 5874, cost 14.55 s 2023-09-05 03:19:08,594 44k INFO ====> Epoch: 5875, cost 14.56 s 2023-09-05 03:19:22,896 44k INFO ====> Epoch: 5876, cost 14.30 s 2023-09-05 03:19:25,769 44k INFO Train Epoch: 5877 [14%] 2023-09-05 03:19:25,769 44k INFO Losses: [2.1717491149902344, 2.8813538551330566, 10.031241416931152, 16.741100311279297, 0.7099849581718445], step: 123400, lr: 4.797232470161961e-05, reference_loss: 32.535430908203125 2023-09-05 03:19:37,658 44k INFO ====> Epoch: 5877, cost 14.76 s 2023-09-05 03:19:52,131 44k INFO ====> Epoch: 5878, cost 14.47 s 2023-09-05 03:20:06,792 44k INFO ====> Epoch: 5879, cost 14.66 s 2023-09-05 03:20:21,310 44k INFO ====> Epoch: 5880, cost 14.52 s 2023-09-05 03:20:35,927 44k INFO ====> Epoch: 5881, cost 14.62 s 2023-09-05 03:20:50,517 44k INFO ====> Epoch: 5882, cost 14.59 s 2023-09-05 03:21:04,921 44k INFO ====> Epoch: 5883, cost 14.40 s 2023-09-05 03:21:19,383 44k INFO ====> Epoch: 5884, cost 14.46 s 2023-09-05 03:21:33,815 44k INFO ====> Epoch: 5885, cost 14.43 s 2023-09-05 03:21:44,403 44k INFO Train Epoch: 5886 [67%] 2023-09-05 03:21:44,404 44k INFO Losses: [2.193037509918213, 2.741231679916382, 9.240985870361328, 16.468746185302734, 0.606075644493103], step: 123600, lr: 4.791838281289394e-05, reference_loss: 31.250076293945312 2023-09-05 03:21:48,624 44k INFO ====> Epoch: 5886, cost 14.81 s 2023-09-05 03:22:03,083 44k INFO ====> Epoch: 5887, cost 14.46 s 2023-09-05 03:22:17,652 44k INFO ====> Epoch: 5888, cost 14.57 s 2023-09-05 03:22:32,146 44k INFO ====> Epoch: 5889, cost 14.49 s 2023-09-05 03:22:46,778 44k INFO ====> Epoch: 5890, cost 14.63 s 2023-09-05 03:23:01,233 44k INFO ====> Epoch: 5891, cost 14.46 s 2023-09-05 03:23:15,735 44k INFO ====> Epoch: 5892, cost 14.50 s 2023-09-05 03:23:30,035 44k INFO ====> Epoch: 5893, cost 14.30 s 2023-09-05 03:23:44,487 44k INFO ====> Epoch: 5894, cost 14.45 s 2023-09-05 03:23:59,298 44k INFO ====> Epoch: 5895, cost 14.81 s 2023-09-05 03:24:02,902 44k INFO Train Epoch: 5896 [19%] 2023-09-05 03:24:02,902 44k INFO Losses: [2.2015087604522705, 2.619173049926758, 9.017427444458008, 14.665608406066895, 0.4899853467941284], step: 123800, lr: 4.78585185157623e-05, reference_loss: 28.993701934814453 2023-09-05 03:24:14,201 44k INFO ====> Epoch: 5896, cost 14.90 s 2023-09-05 03:24:28,849 44k INFO ====> Epoch: 5897, cost 14.65 s 2023-09-05 03:24:43,518 44k INFO ====> Epoch: 5898, cost 14.67 s 2023-09-05 03:24:58,039 44k INFO ====> Epoch: 5899, cost 14.52 s 2023-09-05 03:25:12,594 44k INFO ====> Epoch: 5900, cost 14.56 s 2023-09-05 03:25:26,971 44k INFO ====> Epoch: 5901, cost 14.38 s 2023-09-05 03:25:41,277 44k INFO ====> Epoch: 5902, cost 14.31 s 2023-09-05 03:25:56,108 44k INFO ====> Epoch: 5903, cost 14.83 s 2023-09-05 03:26:10,775 44k INFO ====> Epoch: 5904, cost 14.67 s 2023-09-05 03:26:22,060 44k INFO Train Epoch: 5905 [71%] 2023-09-05 03:26:22,061 44k INFO Losses: [2.248079538345337, 2.5812323093414307, 8.132858276367188, 15.081089973449707, 0.6107205152511597], step: 124000, lr: 4.780470459499839e-05, reference_loss: 28.65397834777832 2023-09-05 03:26:27,462 44k INFO Saving model and optimizer state at iteration 5905 to ./logs\44k\G_124000.pth 2023-09-05 03:26:28,096 44k INFO Saving model and optimizer state at iteration 5905 to ./logs\44k\D_124000.pth 2023-09-05 03:26:29,700 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_121600.pth 2023-09-05 03:26:29,737 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_121600.pth 2023-09-05 03:26:32,926 44k INFO ====> Epoch: 5905, cost 22.15 s 2023-09-05 03:26:47,413 44k INFO ====> Epoch: 5906, cost 14.49 s 2023-09-05 03:27:01,940 44k INFO ====> Epoch: 5907, cost 14.53 s 2023-09-05 03:27:16,379 44k INFO ====> Epoch: 5908, cost 14.44 s 2023-09-05 03:27:30,951 44k INFO ====> Epoch: 5909, cost 14.57 s 2023-09-05 03:27:45,551 44k INFO ====> Epoch: 5910, cost 14.60 s 2023-09-05 03:28:00,190 44k INFO ====> Epoch: 5911, cost 14.64 s 2023-09-05 03:28:14,763 44k INFO ====> Epoch: 5912, cost 14.57 s 2023-09-05 03:28:29,352 44k INFO ====> Epoch: 5913, cost 14.59 s 2023-09-05 03:28:43,970 44k INFO ====> Epoch: 5914, cost 14.62 s 2023-09-05 03:28:48,294 44k INFO Train Epoch: 5915 [24%] 2023-09-05 03:28:48,295 44k INFO Losses: [2.5984156131744385, 2.0492477416992188, 5.930862903594971, 13.526358604431152, 0.6094494462013245], step: 124200, lr: 4.7744982315735775e-05, reference_loss: 24.71433448791504 2023-09-05 03:28:59,092 44k INFO ====> Epoch: 5915, cost 15.12 s 2023-09-05 03:29:13,809 44k INFO ====> Epoch: 5916, cost 14.72 s 2023-09-05 03:29:28,082 44k INFO ====> Epoch: 5917, cost 14.27 s 2023-09-05 03:29:42,652 44k INFO ====> Epoch: 5918, cost 14.57 s 2023-09-05 03:29:57,571 44k INFO ====> Epoch: 5919, cost 14.92 s 2023-09-05 03:30:12,186 44k INFO ====> Epoch: 5920, cost 14.61 s 2023-09-05 03:30:26,749 44k INFO ====> Epoch: 5921, cost 14.56 s 2023-09-05 03:30:41,303 44k INFO ====> Epoch: 5922, cost 14.55 s 2023-09-05 03:30:55,964 44k INFO ====> Epoch: 5923, cost 14.66 s 2023-09-05 03:31:07,943 44k INFO Train Epoch: 5924 [76%] 2023-09-05 03:31:07,943 44k INFO Losses: [2.325256586074829, 2.373528480529785, 9.956304550170898, 16.53006935119629, 0.5251351594924927], step: 124400, lr: 4.769129605935142e-05, reference_loss: 31.71029281616211 2023-09-05 03:31:10,738 44k INFO ====> Epoch: 5924, cost 14.77 s 2023-09-05 03:31:25,201 44k INFO ====> Epoch: 5925, cost 14.46 s 2023-09-05 03:31:39,560 44k INFO ====> Epoch: 5926, cost 14.36 s 2023-09-05 03:31:54,280 44k INFO ====> Epoch: 5927, cost 14.72 s 2023-09-05 03:32:09,046 44k INFO ====> Epoch: 5928, cost 14.77 s 2023-09-05 03:32:23,718 44k INFO ====> Epoch: 5929, cost 14.67 s 2023-09-05 03:32:38,238 44k INFO ====> Epoch: 5930, cost 14.52 s 2023-09-05 03:32:52,894 44k INFO ====> Epoch: 5931, cost 14.66 s 2023-09-05 03:33:07,617 44k INFO ====> Epoch: 5932, cost 14.72 s 2023-09-05 03:33:21,903 44k INFO ====> Epoch: 5933, cost 14.29 s 2023-09-05 03:33:26,754 44k INFO Train Epoch: 5934 [29%] 2023-09-05 03:33:26,754 44k INFO Losses: [2.178022861480713, 2.7338645458221436, 8.297985076904297, 13.763688087463379, 0.4912385046482086], step: 124600, lr: 4.763171546104454e-05, reference_loss: 27.464797973632812 2023-09-05 03:33:36,569 44k INFO ====> Epoch: 5934, cost 14.67 s 2023-09-05 03:33:51,106 44k INFO ====> Epoch: 5935, cost 14.54 s 2023-09-05 03:34:05,778 44k INFO ====> Epoch: 5936, cost 14.67 s 2023-09-05 03:34:20,262 44k INFO ====> Epoch: 5937, cost 14.48 s 2023-09-05 03:34:34,577 44k INFO ====> Epoch: 5938, cost 14.32 s 2023-09-05 03:34:48,970 44k INFO ====> Epoch: 5939, cost 14.39 s 2023-09-05 03:35:03,717 44k INFO ====> Epoch: 5940, cost 14.75 s 2023-09-05 03:35:18,262 44k INFO ====> Epoch: 5941, cost 14.54 s 2023-09-05 03:35:32,543 44k INFO ====> Epoch: 5942, cost 14.28 s 2023-09-05 03:35:45,170 44k INFO Train Epoch: 5943 [81%] 2023-09-05 03:35:45,170 44k INFO Losses: [2.2237839698791504, 2.7962820529937744, 8.653143882751465, 14.14098834991455, 0.6312273740768433], step: 124800, lr: 4.757815656617768e-05, reference_loss: 28.445425033569336 2023-09-05 03:35:50,710 44k INFO Saving model and optimizer state at iteration 5943 to ./logs\44k\G_124800.pth 2023-09-05 03:35:51,342 44k INFO Saving model and optimizer state at iteration 5943 to ./logs\44k\D_124800.pth 2023-09-05 03:35:52,584 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_122400.pth 2023-09-05 03:35:52,625 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_122400.pth 2023-09-05 03:35:54,495 44k INFO ====> Epoch: 5943, cost 21.95 s 2023-09-05 03:36:09,298 44k INFO ====> Epoch: 5944, cost 14.80 s 2023-09-05 03:36:23,788 44k INFO ====> Epoch: 5945, cost 14.49 s 2023-09-05 03:36:38,351 44k INFO ====> Epoch: 5946, cost 14.56 s 2023-09-05 03:36:52,888 44k INFO ====> Epoch: 5947, cost 14.54 s 2023-09-05 03:37:07,453 44k INFO ====> Epoch: 5948, cost 14.57 s 2023-09-05 03:37:21,854 44k INFO ====> Epoch: 5949, cost 14.40 s 2023-09-05 03:37:36,227 44k INFO ====> Epoch: 5950, cost 14.37 s 2023-09-05 03:37:50,781 44k INFO ====> Epoch: 5951, cost 14.55 s 2023-09-05 03:38:05,475 44k INFO ====> Epoch: 5952, cost 14.69 s 2023-09-05 03:38:11,191 44k INFO Train Epoch: 5953 [33%] 2023-09-05 03:38:11,191 44k INFO Losses: [2.153693914413452, 2.7890706062316895, 9.515130996704102, 14.622919082641602, 0.5940881967544556], step: 125000, lr: 4.751871731271258e-05, reference_loss: 29.674901962280273 2023-09-05 03:38:20,251 44k INFO ====> Epoch: 5953, cost 14.78 s 2023-09-05 03:38:34,581 44k INFO ====> Epoch: 5954, cost 14.33 s 2023-09-05 03:38:48,962 44k INFO ====> Epoch: 5955, cost 14.38 s 2023-09-05 03:39:03,579 44k INFO ====> Epoch: 5956, cost 14.62 s 2023-09-05 03:39:18,098 44k INFO ====> Epoch: 5957, cost 14.52 s 2023-09-05 03:39:32,338 44k INFO ====> Epoch: 5958, cost 14.24 s 2023-09-05 03:39:46,732 44k INFO ====> Epoch: 5959, cost 14.39 s 2023-09-05 03:40:01,329 44k INFO ====> Epoch: 5960, cost 14.60 s 2023-09-05 03:40:15,789 44k INFO ====> Epoch: 5961, cost 14.46 s 2023-09-05 03:40:29,122 44k INFO Train Epoch: 5962 [86%] 2023-09-05 03:40:29,122 44k INFO Losses: [2.2025532722473145, 2.6050212383270264, 8.2261381149292, 15.01408863067627, 0.510324239730835], step: 125200, lr: 4.746528547721967e-05, reference_loss: 28.55812644958496 2023-09-05 03:40:30,502 44k INFO ====> Epoch: 5962, cost 14.71 s 2023-09-05 03:40:45,191 44k INFO ====> Epoch: 5963, cost 14.69 s 2023-09-05 03:40:59,833 44k INFO ====> Epoch: 5964, cost 14.64 s 2023-09-05 03:41:14,558 44k INFO ====> Epoch: 5965, cost 14.73 s 2023-09-05 03:41:28,976 44k INFO ====> Epoch: 5966, cost 14.42 s 2023-09-05 03:41:43,395 44k INFO ====> Epoch: 5967, cost 14.42 s 2023-09-05 03:41:58,033 44k INFO ====> Epoch: 5968, cost 14.64 s 2023-09-05 03:42:12,651 44k INFO ====> Epoch: 5969, cost 14.62 s 2023-09-05 03:42:27,129 44k INFO ====> Epoch: 5970, cost 14.48 s 2023-09-05 03:42:41,510 44k INFO ====> Epoch: 5971, cost 14.38 s 2023-09-05 03:42:47,889 44k INFO Train Epoch: 5972 [38%] 2023-09-05 03:42:47,889 44k INFO Losses: [2.06009578704834, 2.8848066329956055, 9.21870231628418, 15.1807279586792, 0.42146626114845276], step: 125400, lr: 4.7405987233279736e-05, reference_loss: 29.76580047607422 2023-09-05 03:42:56,423 44k INFO ====> Epoch: 5972, cost 14.91 s 2023-09-05 03:43:11,143 44k INFO ====> Epoch: 5973, cost 14.72 s 2023-09-05 03:43:25,437 44k INFO ====> Epoch: 5974, cost 14.29 s 2023-09-05 03:43:39,824 44k INFO ====> Epoch: 5975, cost 14.39 s 2023-09-05 03:43:54,357 44k INFO ====> Epoch: 5976, cost 14.53 s 2023-09-05 03:44:08,873 44k INFO ====> Epoch: 5977, cost 14.52 s 2023-09-05 03:44:23,310 44k INFO ====> Epoch: 5978, cost 14.44 s 2023-09-05 03:44:37,821 44k INFO ====> Epoch: 5979, cost 14.51 s 2023-09-05 03:44:52,235 44k INFO ====> Epoch: 5980, cost 14.41 s 2023-09-05 03:45:06,377 44k INFO Train Epoch: 5981 [90%] 2023-09-05 03:45:06,378 44k INFO Losses: [2.331690788269043, 2.373048782348633, 7.958133220672607, 16.19375228881836, 0.7077548503875732], step: 125600, lr: 4.735268215573402e-05, reference_loss: 29.56437873840332 2023-09-05 03:45:11,797 44k INFO Saving model and optimizer state at iteration 5981 to ./logs\44k\G_125600.pth 2023-09-05 03:45:12,391 44k INFO Saving model and optimizer state at iteration 5981 to ./logs\44k\D_125600.pth 2023-09-05 03:45:13,539 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_123200.pth 2023-09-05 03:45:13,577 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_123200.pth 2023-09-05 03:45:14,038 44k INFO ====> Epoch: 5981, cost 21.80 s 2023-09-05 03:45:28,619 44k INFO ====> Epoch: 5982, cost 14.58 s 2023-09-05 03:45:43,185 44k INFO ====> Epoch: 5983, cost 14.57 s 2023-09-05 03:45:57,908 44k INFO ====> Epoch: 5984, cost 14.72 s 2023-09-05 03:46:12,462 44k INFO ====> Epoch: 5985, cost 14.55 s 2023-09-05 03:46:26,883 44k INFO ====> Epoch: 5986, cost 14.42 s 2023-09-05 03:46:41,421 44k INFO ====> Epoch: 5987, cost 14.54 s 2023-09-05 03:46:56,140 44k INFO ====> Epoch: 5988, cost 14.72 s 2023-09-05 03:47:10,775 44k INFO ====> Epoch: 5989, cost 14.64 s 2023-09-05 03:47:25,135 44k INFO ====> Epoch: 5990, cost 14.36 s 2023-09-05 03:47:32,090 44k INFO Train Epoch: 5991 [43%] 2023-09-05 03:47:32,091 44k INFO Losses: [2.2091548442840576, 2.8947064876556396, 9.296579360961914, 15.322101593017578, 0.565434992313385], step: 125800, lr: 4.729352458679812e-05, reference_loss: 30.28797721862793 2023-09-05 03:47:39,714 44k INFO ====> Epoch: 5991, cost 14.58 s 2023-09-05 03:47:54,429 44k INFO ====> Epoch: 5992, cost 14.72 s 2023-09-05 03:48:09,138 44k INFO ====> Epoch: 5993, cost 14.71 s 2023-09-05 03:48:23,662 44k INFO ====> Epoch: 5994, cost 14.53 s 2023-09-05 03:48:38,233 44k INFO ====> Epoch: 5995, cost 14.57 s 2023-09-05 03:48:52,926 44k INFO ====> Epoch: 5996, cost 14.69 s 2023-09-05 03:49:07,475 44k INFO ====> Epoch: 5997, cost 14.55 s 2023-09-05 03:49:21,927 44k INFO ====> Epoch: 5998, cost 14.45 s 2023-09-05 03:49:36,385 44k INFO ====> Epoch: 5999, cost 14.46 s 2023-09-05 03:49:50,998 44k INFO Train Epoch: 6000 [95%] 2023-09-05 03:49:50,999 44k INFO Losses: [2.0299532413482666, 2.8172240257263184, 11.19648551940918, 15.852611541748047, 0.25907984375953674], step: 126000, lr: 4.724034596648789e-05, reference_loss: 32.15535354614258 2023-09-05 03:49:51,262 44k INFO ====> Epoch: 6000, cost 14.88 s 2023-09-05 03:50:05,963 44k INFO ====> Epoch: 6001, cost 14.70 s 2023-09-05 03:50:20,469 44k INFO ====> Epoch: 6002, cost 14.51 s 2023-09-05 03:50:34,846 44k INFO ====> Epoch: 6003, cost 14.38 s 2023-09-05 03:50:49,339 44k INFO ====> Epoch: 6004, cost 14.49 s 2023-09-05 03:51:03,958 44k INFO ====> Epoch: 6005, cost 14.62 s 2023-09-05 03:51:18,480 44k INFO ====> Epoch: 6006, cost 14.52 s 2023-09-05 03:51:32,926 44k INFO ====> Epoch: 6007, cost 14.45 s 2023-09-05 03:51:47,498 44k INFO ====> Epoch: 6008, cost 14.57 s 2023-09-05 03:52:01,962 44k INFO ====> Epoch: 6009, cost 14.46 s 2023-09-05 03:52:09,746 44k INFO Train Epoch: 6010 [48%] 2023-09-05 03:52:09,747 44k INFO Losses: [2.298095226287842, 2.6399967670440674, 9.567070960998535, 15.171693801879883, 0.5542547702789307], step: 126200, lr: 4.718132873882848e-05, reference_loss: 30.231111526489258 2023-09-05 03:52:16,734 44k INFO ====> Epoch: 6010, cost 14.77 s 2023-09-05 03:52:31,239 44k INFO ====> Epoch: 6011, cost 14.50 s 2023-09-05 03:52:45,815 44k INFO ====> Epoch: 6012, cost 14.58 s 2023-09-05 03:53:00,566 44k INFO ====> Epoch: 6013, cost 14.75 s 2023-09-05 03:53:15,067 44k INFO ====> Epoch: 6014, cost 14.50 s 2023-09-05 03:53:29,509 44k INFO ====> Epoch: 6015, cost 14.44 s 2023-09-05 03:53:43,943 44k INFO ====> Epoch: 6016, cost 14.43 s 2023-09-05 03:53:58,541 44k INFO ====> Epoch: 6017, cost 14.60 s 2023-09-05 03:54:13,219 44k INFO ====> Epoch: 6018, cost 14.68 s 2023-09-05 03:54:27,788 44k INFO ====> Epoch: 6019, cost 14.57 s 2023-09-05 03:54:28,515 44k INFO Train Epoch: 6020 [0%] 2023-09-05 03:54:28,515 44k INFO Losses: [2.2281272411346436, 2.692331552505493, 7.836169719696045, 14.208426475524902, 0.5809786319732666], step: 126400, lr: 4.712238524122099e-05, reference_loss: 27.546031951904297 2023-09-05 03:54:33,946 44k INFO Saving model and optimizer state at iteration 6020 to ./logs\44k\G_126400.pth 2023-09-05 03:54:34,554 44k INFO Saving model and optimizer state at iteration 6020 to ./logs\44k\D_126400.pth 2023-09-05 03:54:35,791 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_124000.pth 2023-09-05 03:54:35,825 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_124000.pth 2023-09-05 03:54:49,725 44k INFO ====> Epoch: 6020, cost 21.94 s 2023-09-05 03:55:04,331 44k INFO ====> Epoch: 6021, cost 14.61 s 2023-09-05 03:55:18,727 44k INFO ====> Epoch: 6022, cost 14.40 s 2023-09-05 03:55:33,091 44k INFO ====> Epoch: 6023, cost 14.36 s 2023-09-05 03:55:47,528 44k INFO ====> Epoch: 6024, cost 14.44 s 2023-09-05 03:56:02,193 44k INFO ====> Epoch: 6025, cost 14.66 s 2023-09-05 03:56:16,713 44k INFO ====> Epoch: 6026, cost 14.52 s 2023-09-05 03:56:31,210 44k INFO ====> Epoch: 6027, cost 14.50 s 2023-09-05 03:56:45,598 44k INFO ====> Epoch: 6028, cost 14.39 s 2023-09-05 03:56:54,050 44k INFO Train Epoch: 6029 [52%] 2023-09-05 03:56:54,050 44k INFO Losses: [2.5157103538513184, 2.391097068786621, 7.541003704071045, 14.960360527038574, 0.6474002003669739], step: 126600, lr: 4.706939905643673e-05, reference_loss: 28.055572509765625 2023-09-05 03:57:00,475 44k INFO ====> Epoch: 6029, cost 14.88 s 2023-09-05 03:57:15,072 44k INFO ====> Epoch: 6030, cost 14.60 s 2023-09-05 03:57:29,446 44k INFO ====> Epoch: 6031, cost 14.37 s 2023-09-05 03:57:43,906 44k INFO ====> Epoch: 6032, cost 14.46 s 2023-09-05 03:57:58,393 44k INFO ====> Epoch: 6033, cost 14.49 s 2023-09-05 03:58:13,033 44k INFO ====> Epoch: 6034, cost 14.64 s 2023-09-05 03:58:27,350 44k INFO ====> Epoch: 6035, cost 14.32 s 2023-09-05 03:58:41,802 44k INFO ====> Epoch: 6036, cost 14.45 s 2023-09-05 03:58:56,335 44k INFO ====> Epoch: 6037, cost 14.53 s 2023-09-05 03:59:10,896 44k INFO ====> Epoch: 6038, cost 14.56 s 2023-09-05 03:59:12,343 44k INFO Train Epoch: 6039 [5%] 2023-09-05 03:59:12,344 44k INFO Losses: [2.5430917739868164, 2.5602753162384033, 9.591175079345703, 14.0999755859375, 0.5376067161560059], step: 126800, lr: 4.70105953922579e-05, reference_loss: 29.332124710083008 2023-09-05 03:59:25,547 44k INFO ====> Epoch: 6039, cost 14.65 s 2023-09-05 03:59:40,028 44k INFO ====> Epoch: 6040, cost 14.48 s 2023-09-05 03:59:54,735 44k INFO ====> Epoch: 6041, cost 14.71 s 2023-09-05 04:00:09,409 44k INFO ====> Epoch: 6042, cost 14.67 s 2023-09-05 04:00:23,840 44k INFO ====> Epoch: 6043, cost 14.43 s 2023-09-05 04:00:38,446 44k INFO ====> Epoch: 6044, cost 14.61 s 2023-09-05 04:00:53,043 44k INFO ====> Epoch: 6045, cost 14.60 s 2023-09-05 04:01:07,748 44k INFO ====> Epoch: 6046, cost 14.70 s 2023-09-05 04:01:22,189 44k INFO ====> Epoch: 6047, cost 14.44 s 2023-09-05 04:01:31,264 44k INFO Train Epoch: 6048 [57%] 2023-09-05 04:01:31,264 44k INFO Losses: [2.197643756866455, 2.7468104362487793, 8.973567008972168, 15.335432052612305, 0.49265116453170776], step: 127000, lr: 4.695773490819027e-05, reference_loss: 29.746103286743164 2023-09-05 04:01:36,884 44k INFO ====> Epoch: 6048, cost 14.70 s 2023-09-05 04:01:51,617 44k INFO ====> Epoch: 6049, cost 14.73 s 2023-09-05 04:02:06,197 44k INFO ====> Epoch: 6050, cost 14.58 s 2023-09-05 04:02:20,696 44k INFO ====> Epoch: 6051, cost 14.50 s 2023-09-05 04:02:35,051 44k INFO ====> Epoch: 6052, cost 14.35 s 2023-09-05 04:02:49,619 44k INFO ====> Epoch: 6053, cost 14.57 s 2023-09-05 04:03:04,206 44k INFO ====> Epoch: 6054, cost 14.59 s 2023-09-05 04:03:18,625 44k INFO ====> Epoch: 6055, cost 14.42 s 2023-09-05 04:03:33,218 44k INFO ====> Epoch: 6056, cost 14.59 s 2023-09-05 04:03:47,626 44k INFO ====> Epoch: 6057, cost 14.41 s 2023-09-05 04:03:49,772 44k INFO Train Epoch: 6058 [10%] 2023-09-05 04:03:49,773 44k INFO Losses: [2.164008140563965, 2.7264351844787598, 10.785139083862305, 14.256352424621582, 0.7110545635223389], step: 127200, lr: 4.6899070745709054e-05, reference_loss: 30.642988204956055 2023-09-05 04:03:55,307 44k INFO Saving model and optimizer state at iteration 6058 to ./logs\44k\G_127200.pth 2023-09-05 04:03:56,006 44k INFO Saving model and optimizer state at iteration 6058 to ./logs\44k\D_127200.pth 2023-09-05 04:03:57,129 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_124800.pth 2023-09-05 04:03:57,166 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_124800.pth 2023-09-05 04:04:09,738 44k INFO ====> Epoch: 6058, cost 22.11 s 2023-09-05 04:04:24,164 44k INFO ====> Epoch: 6059, cost 14.43 s 2023-09-05 04:04:38,558 44k INFO ====> Epoch: 6060, cost 14.39 s 2023-09-05 04:04:53,350 44k INFO ====> Epoch: 6061, cost 14.79 s 2023-09-05 04:05:07,955 44k INFO ====> Epoch: 6062, cost 14.60 s 2023-09-05 04:05:22,442 44k INFO ====> Epoch: 6063, cost 14.49 s 2023-09-05 04:05:36,778 44k INFO ====> Epoch: 6064, cost 14.34 s 2023-09-05 04:05:51,435 44k INFO ====> Epoch: 6065, cost 14.66 s 2023-09-05 04:06:06,289 44k INFO ====> Epoch: 6066, cost 14.85 s 2023-09-05 04:06:16,371 44k INFO Train Epoch: 6067 [62%] 2023-09-05 04:06:16,371 44k INFO Losses: [2.2510313987731934, 2.634380340576172, 9.193073272705078, 16.39617156982422, 0.4795549213886261], step: 127400, lr: 4.684633566415447e-05, reference_loss: 30.954212188720703 2023-09-05 04:06:21,385 44k INFO ====> Epoch: 6067, cost 15.10 s 2023-09-05 04:06:35,911 44k INFO ====> Epoch: 6068, cost 14.53 s 2023-09-05 04:06:50,309 44k INFO ====> Epoch: 6069, cost 14.40 s 2023-09-05 04:07:04,923 44k INFO ====> Epoch: 6070, cost 14.61 s 2023-09-05 04:07:19,292 44k INFO ====> Epoch: 6071, cost 14.37 s 2023-09-05 04:07:33,676 44k INFO ====> Epoch: 6072, cost 14.38 s 2023-09-05 04:07:48,338 44k INFO ====> Epoch: 6073, cost 14.66 s 2023-09-05 04:08:02,933 44k INFO ====> Epoch: 6074, cost 14.60 s 2023-09-05 04:08:17,351 44k INFO ====> Epoch: 6075, cost 14.42 s 2023-09-05 04:08:31,768 44k INFO ====> Epoch: 6076, cost 14.42 s 2023-09-05 04:08:34,571 44k INFO Train Epoch: 6077 [14%] 2023-09-05 04:08:34,571 44k INFO Losses: [2.1838979721069336, 2.8142518997192383, 9.070301055908203, 15.97638988494873, 0.5655058026313782], step: 127600, lr: 4.6787810672426816e-05, reference_loss: 30.610347747802734 2023-09-05 04:08:46,511 44k INFO ====> Epoch: 6077, cost 14.74 s 2023-09-05 04:09:01,186 44k INFO ====> Epoch: 6078, cost 14.67 s 2023-09-05 04:09:15,764 44k INFO ====> Epoch: 6079, cost 14.58 s 2023-09-05 04:09:30,039 44k INFO ====> Epoch: 6080, cost 14.27 s 2023-09-05 04:09:44,595 44k INFO ====> Epoch: 6081, cost 14.56 s 2023-09-05 04:09:59,091 44k INFO ====> Epoch: 6082, cost 14.50 s 2023-09-05 04:10:13,591 44k INFO ====> Epoch: 6083, cost 14.50 s 2023-09-05 04:10:28,007 44k INFO ====> Epoch: 6084, cost 14.42 s 2023-09-05 04:10:42,426 44k INFO ====> Epoch: 6085, cost 14.42 s 2023-09-05 04:10:53,121 44k INFO Train Epoch: 6086 [67%] 2023-09-05 04:10:53,122 44k INFO Losses: [2.5373992919921875, 2.2881810665130615, 7.046345233917236, 12.619138717651367, 0.4633466303348541], step: 127800, lr: 4.6735200695889126e-05, reference_loss: 24.954410552978516 2023-09-05 04:10:57,422 44k INFO ====> Epoch: 6086, cost 15.00 s 2023-09-05 04:11:12,000 44k INFO ====> Epoch: 6087, cost 14.58 s 2023-09-05 04:11:26,284 44k INFO ====> Epoch: 6088, cost 14.28 s 2023-09-05 04:11:40,775 44k INFO ====> Epoch: 6089, cost 14.49 s 2023-09-05 04:11:55,524 44k INFO ====> Epoch: 6090, cost 14.75 s 2023-09-05 04:12:10,281 44k INFO ====> Epoch: 6091, cost 14.76 s 2023-09-05 04:12:24,739 44k INFO ====> Epoch: 6092, cost 14.46 s 2023-09-05 04:12:39,225 44k INFO ====> Epoch: 6093, cost 14.49 s 2023-09-05 04:12:53,815 44k INFO ====> Epoch: 6094, cost 14.59 s 2023-09-05 04:13:08,406 44k INFO ====> Epoch: 6095, cost 14.59 s 2023-09-05 04:13:11,915 44k INFO Train Epoch: 6096 [19%] 2023-09-05 04:13:11,916 44k INFO Losses: [2.109591007232666, 3.0935189723968506, 9.77087116241455, 15.660591125488281, 0.7052035331726074], step: 128000, lr: 4.6676814544756065e-05, reference_loss: 31.33977508544922 2023-09-05 04:13:17,567 44k INFO Saving model and optimizer state at iteration 6096 to ./logs\44k\G_128000.pth 2023-09-05 04:13:18,210 44k INFO Saving model and optimizer state at iteration 6096 to ./logs\44k\D_128000.pth 2023-09-05 04:13:19,390 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_125600.pth 2023-09-05 04:13:19,427 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_125600.pth 2023-09-05 04:13:30,312 44k INFO ====> Epoch: 6096, cost 21.91 s 2023-09-05 04:13:44,739 44k INFO ====> Epoch: 6097, cost 14.43 s 2023-09-05 04:13:59,500 44k INFO ====> Epoch: 6098, cost 14.76 s 2023-09-05 04:14:14,217 44k INFO ====> Epoch: 6099, cost 14.72 s 2023-09-05 04:14:28,648 44k INFO ====> Epoch: 6100, cost 14.43 s 2023-09-05 04:14:43,240 44k INFO ====> Epoch: 6101, cost 14.59 s 2023-09-05 04:14:57,865 44k INFO ====> Epoch: 6102, cost 14.62 s 2023-09-05 04:15:12,331 44k INFO ====> Epoch: 6103, cost 14.47 s 2023-09-05 04:15:26,725 44k INFO ====> Epoch: 6104, cost 14.39 s 2023-09-05 04:15:37,910 44k INFO Train Epoch: 6105 [71%] 2023-09-05 04:15:37,910 44k INFO Losses: [2.3164401054382324, 2.4544472694396973, 8.094478607177734, 13.870180130004883, 0.7296475172042847], step: 128200, lr: 4.66243293764449e-05, reference_loss: 27.465192794799805 2023-09-05 04:15:41,371 44k INFO ====> Epoch: 6105, cost 14.65 s 2023-09-05 04:15:55,959 44k INFO ====> Epoch: 6106, cost 14.59 s 2023-09-05 04:16:10,584 44k INFO ====> Epoch: 6107, cost 14.62 s 2023-09-05 04:16:25,059 44k INFO ====> Epoch: 6108, cost 14.48 s 2023-09-05 04:16:39,542 44k INFO ====> Epoch: 6109, cost 14.48 s 2023-09-05 04:16:54,254 44k INFO ====> Epoch: 6110, cost 14.71 s 2023-09-05 04:17:08,864 44k INFO ====> Epoch: 6111, cost 14.61 s 2023-09-05 04:17:23,409 44k INFO ====> Epoch: 6112, cost 14.55 s 2023-09-05 04:17:37,789 44k INFO ====> Epoch: 6113, cost 14.38 s 2023-09-05 04:17:52,406 44k INFO ====> Epoch: 6114, cost 14.62 s 2023-09-05 04:17:56,709 44k INFO Train Epoch: 6115 [24%] 2023-09-05 04:17:56,709 44k INFO Losses: [2.3350446224212646, 2.5671818256378174, 8.804032325744629, 15.445908546447754, 0.5419567227363586], step: 128400, lr: 4.656608173653072e-05, reference_loss: 29.694124221801758 2023-09-05 04:18:07,328 44k INFO ====> Epoch: 6115, cost 14.92 s 2023-09-05 04:18:21,856 44k INFO ====> Epoch: 6116, cost 14.53 s 2023-09-05 04:18:36,230 44k INFO ====> Epoch: 6117, cost 14.37 s 2023-09-05 04:18:50,816 44k INFO ====> Epoch: 6118, cost 14.59 s 2023-09-05 04:19:05,481 44k INFO ====> Epoch: 6119, cost 14.67 s 2023-09-05 04:19:20,064 44k INFO ====> Epoch: 6120, cost 14.58 s 2023-09-05 04:19:34,581 44k INFO ====> Epoch: 6121, cost 14.52 s 2023-09-05 04:19:49,192 44k INFO ====> Epoch: 6122, cost 14.61 s 2023-09-05 04:20:03,910 44k INFO ====> Epoch: 6123, cost 14.72 s 2023-09-05 04:20:15,837 44k INFO Train Epoch: 6124 [76%] 2023-09-05 04:20:15,837 44k INFO Losses: [2.256988525390625, 2.847405195236206, 8.498966217041016, 14.398503303527832, 0.4603535532951355], step: 128600, lr: 4.651372108035977e-05, reference_loss: 28.462217330932617 2023-09-05 04:20:18,620 44k INFO ====> Epoch: 6124, cost 14.71 s 2023-09-05 04:20:33,104 44k INFO ====> Epoch: 6125, cost 14.48 s 2023-09-05 04:20:47,551 44k INFO ====> Epoch: 6126, cost 14.45 s 2023-09-05 04:21:02,104 44k INFO ====> Epoch: 6127, cost 14.55 s 2023-09-05 04:21:16,754 44k INFO ====> Epoch: 6128, cost 14.65 s 2023-09-05 04:21:31,094 44k INFO ====> Epoch: 6129, cost 14.34 s 2023-09-05 04:21:45,745 44k INFO ====> Epoch: 6130, cost 14.65 s 2023-09-05 04:22:00,387 44k INFO ====> Epoch: 6131, cost 14.64 s 2023-09-05 04:22:14,872 44k INFO ====> Epoch: 6132, cost 14.49 s 2023-09-05 04:22:29,208 44k INFO ====> Epoch: 6133, cost 14.34 s 2023-09-05 04:22:34,138 44k INFO Train Epoch: 6134 [29%] 2023-09-05 04:22:34,138 44k INFO Losses: [1.965775966644287, 3.077655792236328, 12.444831848144531, 17.23356819152832, 0.7447977066040039], step: 128800, lr: 4.645561162307015e-05, reference_loss: 35.46662902832031 2023-09-05 04:22:39,601 44k INFO Saving model and optimizer state at iteration 6134 to ./logs\44k\G_128800.pth 2023-09-05 04:22:40,275 44k INFO Saving model and optimizer state at iteration 6134 to ./logs\44k\D_128800.pth 2023-09-05 04:22:41,389 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_126400.pth 2023-09-05 04:22:41,424 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_126400.pth 2023-09-05 04:22:51,111 44k INFO ====> Epoch: 6134, cost 21.90 s 2023-09-05 04:23:05,994 44k INFO ====> Epoch: 6135, cost 14.88 s 2023-09-05 04:23:20,340 44k INFO ====> Epoch: 6136, cost 14.35 s 2023-09-05 04:23:34,686 44k INFO ====> Epoch: 6137, cost 14.35 s 2023-09-05 04:23:49,133 44k INFO ====> Epoch: 6138, cost 14.45 s 2023-09-05 04:24:04,086 44k INFO ====> Epoch: 6139, cost 14.95 s 2023-09-05 04:24:18,662 44k INFO ====> Epoch: 6140, cost 14.58 s 2023-09-05 04:24:33,135 44k INFO ====> Epoch: 6141, cost 14.47 s 2023-09-05 04:24:47,596 44k INFO ====> Epoch: 6142, cost 14.46 s 2023-09-05 04:25:00,451 44k INFO Train Epoch: 6143 [81%] 2023-09-05 04:25:00,451 44k INFO Losses: [2.122565746307373, 2.968320369720459, 10.892468452453613, 16.519107818603516, 0.6653277277946472], step: 129000, lr: 4.640337518365553e-05, reference_loss: 33.16779327392578 2023-09-05 04:25:02,570 44k INFO ====> Epoch: 6143, cost 14.97 s 2023-09-05 04:25:17,145 44k INFO ====> Epoch: 6144, cost 14.57 s 2023-09-05 04:25:31,456 44k INFO ====> Epoch: 6145, cost 14.31 s 2023-09-05 04:25:46,069 44k INFO ====> Epoch: 6146, cost 14.61 s 2023-09-05 04:26:00,715 44k INFO ====> Epoch: 6147, cost 14.65 s 2023-09-05 04:26:15,341 44k INFO ====> Epoch: 6148, cost 14.63 s 2023-09-05 04:26:29,925 44k INFO ====> Epoch: 6149, cost 14.58 s 2023-09-05 04:26:44,394 44k INFO ====> Epoch: 6150, cost 14.47 s 2023-09-05 04:26:59,120 44k INFO ====> Epoch: 6151, cost 14.73 s 2023-09-05 04:27:13,679 44k INFO ====> Epoch: 6152, cost 14.56 s 2023-09-05 04:27:19,311 44k INFO Train Epoch: 6153 [33%] 2023-09-05 04:27:19,312 44k INFO Losses: [2.1956281661987305, 2.8073580265045166, 12.155328750610352, 17.174118041992188, 0.6223526000976562], step: 129200, lr: 4.63454035811757e-05, reference_loss: 34.95478439331055 2023-09-05 04:27:28,344 44k INFO ====> Epoch: 6153, cost 14.66 s 2023-09-05 04:27:42,972 44k INFO ====> Epoch: 6154, cost 14.63 s 2023-09-05 04:27:57,544 44k INFO ====> Epoch: 6155, cost 14.57 s 2023-09-05 04:28:12,162 44k INFO ====> Epoch: 6156, cost 14.62 s 2023-09-05 04:28:26,550 44k INFO ====> Epoch: 6157, cost 14.39 s 2023-09-05 04:28:41,023 44k INFO ====> Epoch: 6158, cost 14.47 s 2023-09-05 04:28:55,734 44k INFO ====> Epoch: 6159, cost 14.71 s 2023-09-05 04:29:10,362 44k INFO ====> Epoch: 6160, cost 14.63 s 2023-09-05 04:29:24,740 44k INFO ====> Epoch: 6161, cost 14.38 s 2023-09-05 04:29:38,029 44k INFO Train Epoch: 6162 [86%] 2023-09-05 04:29:38,030 44k INFO Losses: [2.1444766521453857, 2.834221839904785, 8.410099983215332, 14.254301071166992, 0.6483206748962402], step: 129400, lr: 4.6293291063834254e-05, reference_loss: 28.29142189025879 2023-09-05 04:29:39,417 44k INFO ====> Epoch: 6162, cost 14.68 s 2023-09-05 04:29:54,016 44k INFO ====> Epoch: 6163, cost 14.60 s 2023-09-05 04:30:08,685 44k INFO ====> Epoch: 6164, cost 14.67 s 2023-09-05 04:30:23,268 44k INFO ====> Epoch: 6165, cost 14.58 s 2023-09-05 04:30:37,748 44k INFO ====> Epoch: 6166, cost 14.48 s 2023-09-05 04:30:52,337 44k INFO ====> Epoch: 6167, cost 14.59 s 2023-09-05 04:31:07,221 44k INFO ====> Epoch: 6168, cost 14.88 s 2023-09-05 04:31:21,571 44k INFO ====> Epoch: 6169, cost 14.35 s 2023-09-05 04:31:35,968 44k INFO ====> Epoch: 6170, cost 14.40 s 2023-09-05 04:31:50,708 44k INFO ====> Epoch: 6171, cost 14.74 s 2023-09-05 04:31:57,014 44k INFO Train Epoch: 6172 [38%] 2023-09-05 04:31:57,014 44k INFO Losses: [2.1467788219451904, 2.6223950386047363, 7.9759440422058105, 13.438843727111816, 0.5941482782363892], step: 129600, lr: 4.62354569891271e-05, reference_loss: 26.778108596801758 2023-09-05 04:32:02,473 44k INFO Saving model and optimizer state at iteration 6172 to ./logs\44k\G_129600.pth 2023-09-05 04:32:03,169 44k INFO Saving model and optimizer state at iteration 6172 to ./logs\44k\D_129600.pth 2023-09-05 04:32:04,281 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_127200.pth 2023-09-05 04:32:04,315 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_127200.pth 2023-09-05 04:32:12,489 44k INFO ====> Epoch: 6172, cost 21.78 s 2023-09-05 04:32:26,929 44k INFO ====> Epoch: 6173, cost 14.44 s 2023-09-05 04:32:41,302 44k INFO ====> Epoch: 6174, cost 14.37 s 2023-09-05 04:32:56,031 44k INFO ====> Epoch: 6175, cost 14.73 s 2023-09-05 04:33:10,625 44k INFO ====> Epoch: 6176, cost 14.59 s 2023-09-05 04:33:25,063 44k INFO ====> Epoch: 6177, cost 14.44 s 2023-09-05 04:33:39,398 44k INFO ====> Epoch: 6178, cost 14.34 s 2023-09-05 04:33:53,852 44k INFO ====> Epoch: 6179, cost 14.45 s 2023-09-05 04:34:08,553 44k INFO ====> Epoch: 6180, cost 14.70 s 2023-09-05 04:34:22,664 44k INFO Train Epoch: 6181 [90%] 2023-09-05 04:34:22,664 44k INFO Losses: [2.091158866882324, 2.8427138328552246, 9.51259708404541, 15.920051574707031, 0.7857877016067505], step: 129800, lr: 4.618346809987479e-05, reference_loss: 31.15230941772461 2023-09-05 04:34:23,370 44k INFO ====> Epoch: 6181, cost 14.82 s 2023-09-05 04:34:37,983 44k INFO ====> Epoch: 6182, cost 14.61 s 2023-09-05 04:34:52,614 44k INFO ====> Epoch: 6183, cost 14.63 s 2023-09-05 04:35:07,470 44k INFO ====> Epoch: 6184, cost 14.86 s 2023-09-05 04:35:21,953 44k INFO ====> Epoch: 6185, cost 14.48 s 2023-09-05 04:35:36,467 44k INFO ====> Epoch: 6186, cost 14.51 s 2023-09-05 04:35:51,026 44k INFO ====> Epoch: 6187, cost 14.56 s 2023-09-05 04:36:05,693 44k INFO ====> Epoch: 6188, cost 14.67 s 2023-09-05 04:36:20,145 44k INFO ====> Epoch: 6189, cost 14.45 s 2023-09-05 04:36:34,830 44k INFO ====> Epoch: 6190, cost 14.69 s 2023-09-05 04:36:41,984 44k INFO Train Epoch: 6191 [43%] 2023-09-05 04:36:41,984 44k INFO Losses: [2.272228479385376, 2.7256147861480713, 9.095151901245117, 14.847946166992188, 0.5405977368354797], step: 130000, lr: 4.6125771226679046e-05, reference_loss: 29.481538772583008 2023-09-05 04:36:49,843 44k INFO ====> Epoch: 6191, cost 15.01 s 2023-09-05 04:37:04,330 44k INFO ====> Epoch: 6192, cost 14.49 s 2023-09-05 04:37:18,765 44k INFO ====> Epoch: 6193, cost 14.43 s 2023-09-05 04:37:33,234 44k INFO ====> Epoch: 6194, cost 14.47 s 2023-09-05 04:37:47,691 44k INFO ====> Epoch: 6195, cost 14.46 s 2023-09-05 04:38:02,383 44k INFO ====> Epoch: 6196, cost 14.69 s 2023-09-05 04:38:17,102 44k INFO ====> Epoch: 6197, cost 14.72 s 2023-09-05 04:38:31,597 44k INFO ====> Epoch: 6198, cost 14.49 s 2023-09-05 04:38:46,102 44k INFO ====> Epoch: 6199, cost 14.50 s 2023-09-05 04:39:00,679 44k INFO Train Epoch: 6200 [95%] 2023-09-05 04:39:00,680 44k INFO Losses: [2.323269844055176, 2.706390142440796, 7.197793960571289, 13.878421783447266, 0.5184661149978638], step: 130200, lr: 4.607390567222924e-05, reference_loss: 26.624343872070312 2023-09-05 04:39:00,943 44k INFO ====> Epoch: 6200, cost 14.84 s 2023-09-05 04:39:15,346 44k INFO ====> Epoch: 6201, cost 14.40 s 2023-09-05 04:39:29,723 44k INFO ====> Epoch: 6202, cost 14.38 s 2023-09-05 04:39:44,231 44k INFO ====> Epoch: 6203, cost 14.51 s 2023-09-05 04:39:58,667 44k INFO ====> Epoch: 6204, cost 14.44 s 2023-09-05 04:40:13,160 44k INFO ====> Epoch: 6205, cost 14.49 s 2023-09-05 04:40:27,629 44k INFO ====> Epoch: 6206, cost 14.47 s 2023-09-05 04:40:42,171 44k INFO ====> Epoch: 6207, cost 14.54 s 2023-09-05 04:40:56,692 44k INFO ====> Epoch: 6208, cost 14.52 s 2023-09-05 04:41:11,394 44k INFO ====> Epoch: 6209, cost 14.70 s 2023-09-05 04:41:19,124 44k INFO Train Epoch: 6210 [48%] 2023-09-05 04:41:19,124 44k INFO Losses: [2.0050859451293945, 2.9768166542053223, 11.232454299926758, 16.69565200805664, 0.6586716175079346], step: 130400, lr: 4.601634567505765e-05, reference_loss: 33.56868362426758 2023-09-05 04:41:24,594 44k INFO Saving model and optimizer state at iteration 6210 to ./logs\44k\G_130400.pth 2023-09-05 04:41:25,196 44k INFO Saving model and optimizer state at iteration 6210 to ./logs\44k\D_130400.pth 2023-09-05 04:41:26,777 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_128000.pth 2023-09-05 04:41:26,807 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_128000.pth 2023-09-05 04:41:33,590 44k INFO ====> Epoch: 6210, cost 22.20 s 2023-09-05 04:41:48,220 44k INFO ====> Epoch: 6211, cost 14.63 s 2023-09-05 04:42:02,870 44k INFO ====> Epoch: 6212, cost 14.65 s 2023-09-05 04:42:17,367 44k INFO ====> Epoch: 6213, cost 14.50 s 2023-09-05 04:42:31,824 44k INFO ====> Epoch: 6214, cost 14.46 s 2023-09-05 04:42:46,380 44k INFO ====> Epoch: 6215, cost 14.56 s 2023-09-05 04:43:00,998 44k INFO ====> Epoch: 6216, cost 14.62 s 2023-09-05 04:43:15,586 44k INFO ====> Epoch: 6217, cost 14.59 s 2023-09-05 04:43:30,006 44k INFO ====> Epoch: 6218, cost 14.42 s 2023-09-05 04:43:44,372 44k INFO ====> Epoch: 6219, cost 14.37 s 2023-09-05 04:43:45,071 44k INFO Train Epoch: 6220 [0%] 2023-09-05 04:43:45,071 44k INFO Losses: [2.230069637298584, 2.786207914352417, 8.700770378112793, 15.127523422241211, 0.5509953498840332], step: 130600, lr: 4.5958857587424146e-05, reference_loss: 29.395565032958984 2023-09-05 04:43:59,329 44k INFO ====> Epoch: 6220, cost 14.96 s 2023-09-05 04:44:13,996 44k INFO ====> Epoch: 6221, cost 14.67 s 2023-09-05 04:44:28,540 44k INFO ====> Epoch: 6222, cost 14.54 s 2023-09-05 04:44:43,036 44k INFO ====> Epoch: 6223, cost 14.50 s 2023-09-05 04:44:57,495 44k INFO ====> Epoch: 6224, cost 14.46 s 2023-09-05 04:45:12,024 44k INFO ====> Epoch: 6225, cost 14.53 s 2023-09-05 04:45:26,509 44k INFO ====> Epoch: 6226, cost 14.48 s 2023-09-05 04:45:40,928 44k INFO ====> Epoch: 6227, cost 14.42 s 2023-09-05 04:45:55,570 44k INFO ====> Epoch: 6228, cost 14.64 s 2023-09-05 04:46:03,963 44k INFO Train Epoch: 6229 [52%] 2023-09-05 04:46:03,963 44k INFO Losses: [1.8925174474716187, 3.271327495574951, 8.994064331054688, 15.331314086914062, 0.6780794858932495], step: 130800, lr: 4.590717971695695e-05, reference_loss: 30.16730308532715 2023-09-05 04:46:10,476 44k INFO ====> Epoch: 6229, cost 14.91 s 2023-09-05 04:46:24,962 44k INFO ====> Epoch: 6230, cost 14.49 s 2023-09-05 04:46:39,359 44k INFO ====> Epoch: 6231, cost 14.40 s 2023-09-05 04:46:53,837 44k INFO ====> Epoch: 6232, cost 14.48 s 2023-09-05 04:47:08,558 44k INFO ====> Epoch: 6233, cost 14.72 s 2023-09-05 04:47:23,000 44k INFO ====> Epoch: 6234, cost 14.44 s 2023-09-05 04:47:37,283 44k INFO ====> Epoch: 6235, cost 14.28 s 2023-09-05 04:47:51,872 44k INFO ====> Epoch: 6236, cost 14.59 s 2023-09-05 04:48:06,522 44k INFO ====> Epoch: 6237, cost 14.65 s 2023-09-05 04:48:20,982 44k INFO ====> Epoch: 6238, cost 14.46 s 2023-09-05 04:48:22,399 44k INFO Train Epoch: 6239 [5%] 2023-09-05 04:48:22,400 44k INFO Losses: [2.063563823699951, 2.9770824909210205, 9.090275764465332, 15.705869674682617, 0.5976365804672241], step: 131000, lr: 4.584982801003933e-05, reference_loss: 30.434429168701172 2023-09-05 04:48:35,827 44k INFO ====> Epoch: 6239, cost 14.84 s 2023-09-05 04:48:50,333 44k INFO ====> Epoch: 6240, cost 14.51 s 2023-09-05 04:49:04,789 44k INFO ====> Epoch: 6241, cost 14.46 s 2023-09-05 04:49:19,337 44k INFO ====> Epoch: 6242, cost 14.55 s 2023-09-05 04:49:33,507 44k INFO ====> Epoch: 6243, cost 14.17 s 2023-09-05 04:49:47,934 44k INFO ====> Epoch: 6244, cost 14.43 s 2023-09-05 04:50:02,681 44k INFO ====> Epoch: 6245, cost 14.75 s 2023-09-05 04:50:17,182 44k INFO ====> Epoch: 6246, cost 14.50 s 2023-09-05 04:50:31,678 44k INFO ====> Epoch: 6247, cost 14.50 s 2023-09-05 04:50:40,818 44k INFO Train Epoch: 6248 [57%] 2023-09-05 04:50:40,819 44k INFO Losses: [2.4664900302886963, 2.561906099319458, 8.65649127960205, 15.075240135192871, 0.7635936141014099], step: 131200, lr: 4.579827273653545e-05, reference_loss: 29.52372169494629 2023-09-05 04:50:46,282 44k INFO Saving model and optimizer state at iteration 6248 to ./logs\44k\G_131200.pth 2023-09-05 04:50:46,969 44k INFO Saving model and optimizer state at iteration 6248 to ./logs\44k\D_131200.pth 2023-09-05 04:50:48,556 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_128800.pth 2023-09-05 04:50:48,591 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_128800.pth 2023-09-05 04:50:53,974 44k INFO ====> Epoch: 6248, cost 22.30 s 2023-09-05 04:51:08,457 44k INFO ====> Epoch: 6249, cost 14.48 s 2023-09-05 04:51:22,945 44k INFO ====> Epoch: 6250, cost 14.49 s 2023-09-05 04:51:37,345 44k INFO ====> Epoch: 6251, cost 14.40 s 2023-09-05 04:51:51,974 44k INFO ====> Epoch: 6252, cost 14.63 s 2023-09-05 04:52:06,582 44k INFO ====> Epoch: 6253, cost 14.61 s 2023-09-05 04:52:21,274 44k INFO ====> Epoch: 6254, cost 14.69 s 2023-09-05 04:52:35,629 44k INFO ====> Epoch: 6255, cost 14.35 s 2023-09-05 04:52:50,254 44k INFO ====> Epoch: 6256, cost 14.63 s 2023-09-05 04:53:04,830 44k INFO ====> Epoch: 6257, cost 14.58 s 2023-09-05 04:53:06,974 44k INFO Train Epoch: 6258 [10%] 2023-09-05 04:53:06,974 44k INFO Losses: [2.2323126792907715, 2.611856460571289, 10.228365898132324, 15.781757354736328, 0.6433414220809937], step: 131400, lr: 4.574105708679365e-05, reference_loss: 31.49763298034668 2023-09-05 04:53:19,700 44k INFO ====> Epoch: 6258, cost 14.87 s 2023-09-05 04:53:34,160 44k INFO ====> Epoch: 6259, cost 14.46 s 2023-09-05 04:53:48,871 44k INFO ====> Epoch: 6260, cost 14.71 s 2023-09-05 04:54:03,857 44k INFO ====> Epoch: 6261, cost 14.99 s 2023-09-05 04:54:18,463 44k INFO ====> Epoch: 6262, cost 14.61 s 2023-09-05 04:54:32,905 44k INFO ====> Epoch: 6263, cost 14.44 s 2023-09-05 04:54:47,442 44k INFO ====> Epoch: 6264, cost 14.54 s 2023-09-05 04:55:01,885 44k INFO ====> Epoch: 6265, cost 14.44 s 2023-09-05 04:55:16,358 44k INFO ====> Epoch: 6266, cost 14.47 s 2023-09-05 04:55:26,071 44k INFO Train Epoch: 6267 [62%] 2023-09-05 04:55:26,072 44k INFO Losses: [2.1922061443328857, 2.7149555683135986, 8.520379066467285, 16.354232788085938, 0.5715298056602478], step: 131600, lr: 4.5689624119412614e-05, reference_loss: 30.353302001953125 2023-09-05 04:55:30,871 44k INFO ====> Epoch: 6267, cost 14.51 s 2023-09-05 04:55:45,356 44k INFO ====> Epoch: 6268, cost 14.48 s 2023-09-05 04:55:59,908 44k INFO ====> Epoch: 6269, cost 14.55 s 2023-09-05 04:56:14,426 44k INFO ====> Epoch: 6270, cost 14.52 s 2023-09-05 04:56:28,991 44k INFO ====> Epoch: 6271, cost 14.56 s 2023-09-05 04:56:43,454 44k INFO ====> Epoch: 6272, cost 14.46 s 2023-09-05 04:56:58,065 44k INFO ====> Epoch: 6273, cost 14.61 s 2023-09-05 04:57:12,774 44k INFO ====> Epoch: 6274, cost 14.71 s 2023-09-05 04:57:27,174 44k INFO ====> Epoch: 6275, cost 14.40 s 2023-09-05 04:57:41,582 44k INFO ====> Epoch: 6276, cost 14.41 s 2023-09-05 04:57:44,471 44k INFO Train Epoch: 6277 [14%] 2023-09-05 04:57:44,472 44k INFO Losses: [2.1598548889160156, 2.765793800354004, 9.242266654968262, 15.159499168395996, 0.5677414536476135], step: 131800, lr: 4.5632544204074124e-05, reference_loss: 29.895156860351562 2023-09-05 04:57:56,403 44k INFO ====> Epoch: 6277, cost 14.82 s 2023-09-05 04:58:10,992 44k INFO ====> Epoch: 6278, cost 14.59 s 2023-09-05 04:58:25,522 44k INFO ====> Epoch: 6279, cost 14.53 s 2023-09-05 04:58:39,840 44k INFO ====> Epoch: 6280, cost 14.32 s 2023-09-05 04:58:54,437 44k INFO ====> Epoch: 6281, cost 14.60 s 2023-09-05 04:59:09,117 44k INFO ====> Epoch: 6282, cost 14.68 s 2023-09-05 04:59:23,425 44k INFO ====> Epoch: 6283, cost 14.31 s 2023-09-05 04:59:37,692 44k INFO ====> Epoch: 6284, cost 14.27 s 2023-09-05 04:59:52,427 44k INFO ====> Epoch: 6285, cost 14.73 s 2023-09-05 05:00:03,180 44k INFO Train Epoch: 6286 [67%] 2023-09-05 05:00:03,181 44k INFO Losses: [2.4217169284820557, 2.5738091468811035, 9.019325256347656, 15.473443984985352, 0.7018595933914185], step: 132000, lr: 4.5581233252665456e-05, reference_loss: 30.190156936645508 2023-09-05 05:00:08,631 44k INFO Saving model and optimizer state at iteration 6286 to ./logs\44k\G_132000.pth 2023-09-05 05:00:09,382 44k INFO Saving model and optimizer state at iteration 6286 to ./logs\44k\D_132000.pth 2023-09-05 05:00:10,956 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_129600.pth 2023-09-05 05:00:10,989 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_129600.pth 2023-09-05 05:00:15,036 44k INFO ====> Epoch: 6286, cost 22.61 s 2023-09-05 05:00:29,587 44k INFO ====> Epoch: 6287, cost 14.55 s 2023-09-05 05:00:44,088 44k INFO ====> Epoch: 6288, cost 14.50 s 2023-09-05 05:00:58,628 44k INFO ====> Epoch: 6289, cost 14.54 s 2023-09-05 05:01:13,149 44k INFO ====> Epoch: 6290, cost 14.52 s 2023-09-05 05:01:27,628 44k INFO ====> Epoch: 6291, cost 14.48 s 2023-09-05 05:01:41,829 44k INFO ====> Epoch: 6292, cost 14.20 s 2023-09-05 05:01:56,354 44k INFO ====> Epoch: 6293, cost 14.52 s 2023-09-05 05:02:10,931 44k INFO ====> Epoch: 6294, cost 14.58 s 2023-09-05 05:02:25,356 44k INFO ====> Epoch: 6295, cost 14.42 s 2023-09-05 05:02:28,811 44k INFO Train Epoch: 6296 [19%] 2023-09-05 05:02:28,811 44k INFO Losses: [2.3223814964294434, 2.4103593826293945, 8.003429412841797, 15.715252876281738, 0.6488956212997437], step: 132200, lr: 4.552428874972347e-05, reference_loss: 29.100318908691406 2023-09-05 05:02:40,207 44k INFO ====> Epoch: 6296, cost 14.85 s 2023-09-05 05:02:54,726 44k INFO ====> Epoch: 6297, cost 14.52 s 2023-09-05 05:03:09,279 44k INFO ====> Epoch: 6298, cost 14.55 s 2023-09-05 05:03:23,584 44k INFO ====> Epoch: 6299, cost 14.30 s 2023-09-05 05:03:37,871 44k INFO ====> Epoch: 6300, cost 14.29 s 2023-09-05 05:03:52,357 44k INFO ====> Epoch: 6301, cost 14.49 s 2023-09-05 05:04:06,944 44k INFO ====> Epoch: 6302, cost 14.59 s 2023-09-05 05:04:21,421 44k INFO ====> Epoch: 6303, cost 14.48 s 2023-09-05 05:04:35,938 44k INFO ====> Epoch: 6304, cost 14.52 s 2023-09-05 05:04:47,116 44k INFO Train Epoch: 6305 [71%] 2023-09-05 05:04:47,117 44k INFO Losses: [2.303800344467163, 2.7242014408111572, 8.747452735900879, 14.135374069213867, 0.5553852915763855], step: 132400, lr: 4.5473099524825e-05, reference_loss: 28.46621322631836 2023-09-05 05:04:50,656 44k INFO ====> Epoch: 6305, cost 14.72 s 2023-09-05 05:05:05,160 44k INFO ====> Epoch: 6306, cost 14.50 s 2023-09-05 05:05:19,673 44k INFO ====> Epoch: 6307, cost 14.51 s 2023-09-05 05:05:34,114 44k INFO ====> Epoch: 6308, cost 14.44 s 2023-09-05 05:05:48,494 44k INFO ====> Epoch: 6309, cost 14.38 s 2023-09-05 05:06:03,078 44k INFO ====> Epoch: 6310, cost 14.58 s 2023-09-05 05:06:17,670 44k INFO ====> Epoch: 6311, cost 14.59 s 2023-09-05 05:06:32,266 44k INFO ====> Epoch: 6312, cost 14.60 s 2023-09-05 05:06:46,954 44k INFO ====> Epoch: 6313, cost 14.69 s 2023-09-05 05:07:01,573 44k INFO ====> Epoch: 6314, cost 14.62 s 2023-09-05 05:07:05,824 44k INFO Train Epoch: 6315 [24%] 2023-09-05 05:07:05,825 44k INFO Losses: [1.963306188583374, 3.0130293369293213, 9.852287292480469, 13.266708374023438, 0.5799580216407776], step: 132600, lr: 4.5416290113036635e-05, reference_loss: 28.675289154052734 2023-09-05 05:07:16,289 44k INFO ====> Epoch: 6315, cost 14.72 s 2023-09-05 05:07:30,669 44k INFO ====> Epoch: 6316, cost 14.38 s 2023-09-05 05:07:45,156 44k INFO ====> Epoch: 6317, cost 14.49 s 2023-09-05 05:07:59,831 44k INFO ====> Epoch: 6318, cost 14.68 s 2023-09-05 05:08:14,303 44k INFO ====> Epoch: 6319, cost 14.47 s 2023-09-05 05:08:28,978 44k INFO ====> Epoch: 6320, cost 14.67 s 2023-09-05 05:08:43,427 44k INFO ====> Epoch: 6321, cost 14.45 s 2023-09-05 05:08:57,958 44k INFO ====> Epoch: 6322, cost 14.53 s 2023-09-05 05:09:12,683 44k INFO ====> Epoch: 6323, cost 14.72 s 2023-09-05 05:09:24,479 44k INFO Train Epoch: 6324 [76%] 2023-09-05 05:09:24,479 44k INFO Losses: [2.185439348220825, 2.590740203857422, 10.777626037597656, 17.379976272583008, 0.6823241114616394], step: 132800, lr: 4.536522232587292e-05, reference_loss: 33.61610412597656 2023-09-05 05:09:29,994 44k INFO Saving model and optimizer state at iteration 6324 to ./logs\44k\G_132800.pth 2023-09-05 05:09:30,602 44k INFO Saving model and optimizer state at iteration 6324 to ./logs\44k\D_132800.pth 2023-09-05 05:09:31,837 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_130400.pth 2023-09-05 05:09:31,872 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_130400.pth 2023-09-05 05:09:34,429 44k INFO ====> Epoch: 6324, cost 21.75 s 2023-09-05 05:09:48,812 44k INFO ====> Epoch: 6325, cost 14.38 s 2023-09-05 05:10:03,331 44k INFO ====> Epoch: 6326, cost 14.52 s 2023-09-05 05:10:17,827 44k INFO ====> Epoch: 6327, cost 14.50 s 2023-09-05 05:10:32,180 44k INFO ====> Epoch: 6328, cost 14.35 s 2023-09-05 05:10:46,819 44k INFO ====> Epoch: 6329, cost 14.64 s 2023-09-05 05:11:01,417 44k INFO ====> Epoch: 6330, cost 14.60 s 2023-09-05 05:11:16,011 44k INFO ====> Epoch: 6331, cost 14.59 s 2023-09-05 05:11:30,443 44k INFO ====> Epoch: 6332, cost 14.43 s 2023-09-05 05:11:44,852 44k INFO ====> Epoch: 6333, cost 14.41 s 2023-09-05 05:11:49,834 44k INFO Train Epoch: 6334 [29%] 2023-09-05 05:11:49,834 44k INFO Losses: [2.228062868118286, 2.693371295928955, 8.874898910522461, 14.393054008483887, 0.5583049654960632], step: 133000, lr: 4.530854768475735e-05, reference_loss: 28.74769401550293 2023-09-05 05:11:59,791 44k INFO ====> Epoch: 6334, cost 14.94 s 2023-09-05 05:12:14,555 44k INFO ====> Epoch: 6335, cost 14.76 s 2023-09-05 05:12:29,008 44k INFO ====> Epoch: 6336, cost 14.45 s 2023-09-05 05:12:43,425 44k INFO ====> Epoch: 6337, cost 14.42 s 2023-09-05 05:12:57,901 44k INFO ====> Epoch: 6338, cost 14.48 s 2023-09-05 05:13:12,548 44k INFO ====> Epoch: 6339, cost 14.65 s 2023-09-05 05:13:27,029 44k INFO ====> Epoch: 6340, cost 14.48 s 2023-09-05 05:13:41,493 44k INFO ====> Epoch: 6341, cost 14.46 s 2023-09-05 05:13:56,316 44k INFO ====> Epoch: 6342, cost 14.82 s 2023-09-05 05:14:09,300 44k INFO Train Epoch: 6343 [81%] 2023-09-05 05:14:09,300 44k INFO Losses: [2.056411027908325, 3.095311403274536, 10.469067573547363, 14.711308479309082, 0.5668825507164001], step: 133200, lr: 4.525760104723801e-05, reference_loss: 30.898983001708984 2023-09-05 05:14:11,383 44k INFO ====> Epoch: 6343, cost 15.07 s 2023-09-05 05:14:25,692 44k INFO ====> Epoch: 6344, cost 14.31 s 2023-09-05 05:14:40,165 44k INFO ====> Epoch: 6345, cost 14.47 s 2023-09-05 05:14:54,609 44k INFO ====> Epoch: 6346, cost 14.44 s 2023-09-05 05:15:09,297 44k INFO ====> Epoch: 6347, cost 14.69 s 2023-09-05 05:15:23,729 44k INFO ====> Epoch: 6348, cost 14.43 s 2023-09-05 05:15:38,175 44k INFO ====> Epoch: 6349, cost 14.45 s 2023-09-05 05:15:52,693 44k INFO ====> Epoch: 6350, cost 14.52 s 2023-09-05 05:16:07,261 44k INFO ====> Epoch: 6351, cost 14.57 s 2023-09-05 05:16:21,670 44k INFO ====> Epoch: 6352, cost 14.41 s 2023-09-05 05:16:27,243 44k INFO Train Epoch: 6353 [33%] 2023-09-05 05:16:27,243 44k INFO Losses: [2.1486682891845703, 2.8874101638793945, 8.655996322631836, 15.081416130065918, 0.5710264444351196], step: 133400, lr: 4.520106085707475e-05, reference_loss: 29.34451675415039 2023-09-05 05:16:36,482 44k INFO ====> Epoch: 6353, cost 14.81 s 2023-09-05 05:16:51,185 44k INFO ====> Epoch: 6354, cost 14.70 s 2023-09-05 05:17:05,839 44k INFO ====> Epoch: 6355, cost 14.65 s 2023-09-05 05:17:20,417 44k INFO ====> Epoch: 6356, cost 14.58 s 2023-09-05 05:17:34,714 44k INFO ====> Epoch: 6357, cost 14.30 s 2023-09-05 05:17:49,253 44k INFO ====> Epoch: 6358, cost 14.54 s 2023-09-05 05:18:04,187 44k INFO ====> Epoch: 6359, cost 14.93 s 2023-09-05 05:18:18,708 44k INFO ====> Epoch: 6360, cost 14.52 s 2023-09-05 05:18:33,362 44k INFO ====> Epoch: 6361, cost 14.65 s 2023-09-05 05:18:46,733 44k INFO Train Epoch: 6362 [86%] 2023-09-05 05:18:46,733 44k INFO Losses: [2.2211341857910156, 2.600506544113159, 9.630736351013184, 15.342936515808105, 0.6180669665336609], step: 133600, lr: 4.515023508179284e-05, reference_loss: 30.413381576538086 2023-09-05 05:18:52,191 44k INFO Saving model and optimizer state at iteration 6362 to ./logs\44k\G_133600.pth 2023-09-05 05:18:53,225 44k INFO Saving model and optimizer state at iteration 6362 to ./logs\44k\D_133600.pth 2023-09-05 05:18:54,419 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_131200.pth 2023-09-05 05:18:54,453 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_131200.pth 2023-09-05 05:18:55,627 44k INFO ====> Epoch: 6362, cost 22.26 s 2023-09-05 05:19:10,168 44k INFO ====> Epoch: 6363, cost 14.54 s 2023-09-05 05:19:24,506 44k INFO ====> Epoch: 6364, cost 14.34 s 2023-09-05 05:19:38,930 44k INFO ====> Epoch: 6365, cost 14.42 s 2023-09-05 05:19:53,542 44k INFO ====> Epoch: 6366, cost 14.61 s 2023-09-05 05:20:08,271 44k INFO ====> Epoch: 6367, cost 14.73 s 2023-09-05 05:20:22,741 44k INFO ====> Epoch: 6368, cost 14.47 s 2023-09-05 05:20:37,289 44k INFO ====> Epoch: 6369, cost 14.55 s 2023-09-05 05:20:51,766 44k INFO ====> Epoch: 6370, cost 14.48 s 2023-09-05 05:21:06,267 44k INFO ====> Epoch: 6371, cost 14.50 s 2023-09-05 05:21:12,596 44k INFO Train Epoch: 6372 [38%] 2023-09-05 05:21:12,597 44k INFO Losses: [2.2682039737701416, 2.6745429039001465, 10.044501304626465, 16.053436279296875, 0.5629950761795044], step: 133800, lr: 4.509382902361984e-05, reference_loss: 31.603679656982422 2023-09-05 05:21:20,983 44k INFO ====> Epoch: 6372, cost 14.72 s 2023-09-05 05:21:35,407 44k INFO ====> Epoch: 6373, cost 14.42 s 2023-09-05 05:21:49,924 44k INFO ====> Epoch: 6374, cost 14.52 s 2023-09-05 05:22:04,589 44k INFO ====> Epoch: 6375, cost 14.66 s 2023-09-05 05:22:19,118 44k INFO ====> Epoch: 6376, cost 14.53 s 2023-09-05 05:22:33,682 44k INFO ====> Epoch: 6377, cost 14.56 s 2023-09-05 05:22:48,348 44k INFO ====> Epoch: 6378, cost 14.67 s 2023-09-05 05:23:03,050 44k INFO ====> Epoch: 6379, cost 14.70 s 2023-09-05 05:23:17,679 44k INFO ====> Epoch: 6380, cost 14.63 s 2023-09-05 05:23:31,576 44k INFO Train Epoch: 6381 [90%] 2023-09-05 05:23:31,577 44k INFO Losses: [2.30884051322937, 2.506439685821533, 8.255531311035156, 15.577583312988281, 0.5010294914245605], step: 134000, lr: 4.504312382385026e-05, reference_loss: 29.149425506591797 2023-09-05 05:23:32,332 44k INFO ====> Epoch: 6381, cost 14.65 s 2023-09-05 05:23:46,978 44k INFO ====> Epoch: 6382, cost 14.65 s 2023-09-05 05:24:01,841 44k INFO ====> Epoch: 6383, cost 14.86 s 2023-09-05 05:24:16,557 44k INFO ====> Epoch: 6384, cost 14.72 s 2023-09-05 05:24:30,955 44k INFO ====> Epoch: 6385, cost 14.40 s 2023-09-05 05:24:45,521 44k INFO ====> Epoch: 6386, cost 14.57 s 2023-09-05 05:25:00,148 44k INFO ====> Epoch: 6387, cost 14.63 s 2023-09-05 05:25:14,641 44k INFO ====> Epoch: 6388, cost 14.49 s 2023-09-05 05:25:28,974 44k INFO ====> Epoch: 6389, cost 14.33 s 2023-09-05 05:25:43,508 44k INFO ====> Epoch: 6390, cost 14.53 s 2023-09-05 05:25:50,709 44k INFO Train Epoch: 6391 [43%] 2023-09-05 05:25:50,710 44k INFO Losses: [2.092849016189575, 2.8853771686553955, 9.211339950561523, 14.395040512084961, 0.5355775356292725], step: 134200, lr: 4.498685157946219e-05, reference_loss: 29.12018394470215 2023-09-05 05:25:58,620 44k INFO ====> Epoch: 6391, cost 15.11 s 2023-09-05 05:26:13,025 44k INFO ====> Epoch: 6392, cost 14.41 s 2023-09-05 05:26:27,688 44k INFO ====> Epoch: 6393, cost 14.66 s 2023-09-05 05:26:42,210 44k INFO ====> Epoch: 6394, cost 14.52 s 2023-09-05 05:26:56,865 44k INFO ====> Epoch: 6395, cost 14.66 s 2023-09-05 05:27:11,401 44k INFO ====> Epoch: 6396, cost 14.53 s 2023-09-05 05:27:25,757 44k INFO ====> Epoch: 6397, cost 14.36 s 2023-09-05 05:27:40,128 44k INFO ====> Epoch: 6398, cost 14.37 s 2023-09-05 05:27:54,782 44k INFO ====> Epoch: 6399, cost 14.65 s 2023-09-05 05:28:09,470 44k INFO Train Epoch: 6400 [95%] 2023-09-05 05:28:09,470 44k INFO Losses: [2.1246519088745117, 2.668114423751831, 8.57209300994873, 15.844152450561523, 0.30014246702194214], step: 134400, lr: 4.4936266669160025e-05, reference_loss: 29.509153366088867 2023-09-05 05:28:14,910 44k INFO Saving model and optimizer state at iteration 6400 to ./logs\44k\G_134400.pth 2023-09-05 05:28:15,587 44k INFO Saving model and optimizer state at iteration 6400 to ./logs\44k\D_134400.pth 2023-09-05 05:28:16,666 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_132000.pth 2023-09-05 05:28:16,699 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_132000.pth 2023-09-05 05:28:16,699 44k INFO ====> Epoch: 6400, cost 21.92 s 2023-09-05 05:28:31,291 44k INFO ====> Epoch: 6401, cost 14.59 s 2023-09-05 05:28:45,792 44k INFO ====> Epoch: 6402, cost 14.50 s 2023-09-05 05:29:00,495 44k INFO ====> Epoch: 6403, cost 14.70 s 2023-09-05 05:29:14,904 44k INFO ====> Epoch: 6404, cost 14.41 s 2023-09-05 05:29:29,380 44k INFO ====> Epoch: 6405, cost 14.48 s 2023-09-05 05:29:43,855 44k INFO ====> Epoch: 6406, cost 14.47 s 2023-09-05 05:29:58,545 44k INFO ====> Epoch: 6407, cost 14.69 s 2023-09-05 05:30:13,161 44k INFO ====> Epoch: 6408, cost 14.62 s 2023-09-05 05:30:27,623 44k INFO ====> Epoch: 6409, cost 14.46 s 2023-09-05 05:30:35,371 44k INFO Train Epoch: 6410 [48%] 2023-09-05 05:30:35,372 44k INFO Losses: [2.1876540184020996, 2.654465675354004, 10.28619384765625, 16.22835922241211, 0.6887229084968567], step: 134600, lr: 4.488012792110642e-05, reference_loss: 32.04539489746094 2023-09-05 05:30:42,454 44k INFO ====> Epoch: 6410, cost 14.83 s 2023-09-05 05:30:57,186 44k INFO ====> Epoch: 6411, cost 14.73 s 2023-09-05 05:31:11,789 44k INFO ====> Epoch: 6412, cost 14.60 s 2023-09-05 05:31:26,276 44k INFO ====> Epoch: 6413, cost 14.49 s 2023-09-05 05:31:40,837 44k INFO ====> Epoch: 6414, cost 14.56 s 2023-09-05 05:31:55,486 44k INFO ====> Epoch: 6415, cost 14.65 s 2023-09-05 05:32:10,101 44k INFO ====> Epoch: 6416, cost 14.62 s 2023-09-05 05:32:24,551 44k INFO ====> Epoch: 6417, cost 14.45 s 2023-09-05 05:32:38,972 44k INFO ====> Epoch: 6418, cost 14.42 s 2023-09-05 05:32:53,616 44k INFO ====> Epoch: 6419, cost 14.64 s 2023-09-05 05:32:54,265 44k INFO Train Epoch: 6420 [0%] 2023-09-05 05:32:54,266 44k INFO Losses: [2.3473880290985107, 2.5180554389953613, 8.740460395812988, 14.48512077331543, 0.5886098146438599], step: 134800, lr: 4.482405930702848e-05, reference_loss: 28.67963409423828 2023-09-05 05:33:08,544 44k INFO ====> Epoch: 6420, cost 14.93 s 2023-09-05 05:33:22,935 44k INFO ====> Epoch: 6421, cost 14.39 s 2023-09-05 05:33:37,470 44k INFO ====> Epoch: 6422, cost 14.53 s 2023-09-05 05:33:52,092 44k INFO ====> Epoch: 6423, cost 14.62 s 2023-09-05 05:34:06,630 44k INFO ====> Epoch: 6424, cost 14.54 s 2023-09-05 05:34:21,105 44k INFO ====> Epoch: 6425, cost 14.48 s 2023-09-05 05:34:35,589 44k INFO ====> Epoch: 6426, cost 14.48 s 2023-09-05 05:34:50,048 44k INFO ====> Epoch: 6427, cost 14.46 s 2023-09-05 05:35:04,656 44k INFO ====> Epoch: 6428, cost 14.61 s 2023-09-05 05:35:13,154 44k INFO Train Epoch: 6429 [52%] 2023-09-05 05:35:13,154 44k INFO Losses: [2.134542942047119, 2.746661424636841, 8.90135669708252, 15.298355102539062, 0.477598637342453], step: 135000, lr: 4.477365744648885e-05, reference_loss: 29.558515548706055 2023-09-05 05:35:19,476 44k INFO ====> Epoch: 6429, cost 14.82 s 2023-09-05 05:35:33,721 44k INFO ====> Epoch: 6430, cost 14.24 s 2023-09-05 05:35:48,519 44k INFO ====> Epoch: 6431, cost 14.80 s 2023-09-05 05:36:03,270 44k INFO ====> Epoch: 6432, cost 14.75 s 2023-09-05 05:36:17,802 44k INFO ====> Epoch: 6433, cost 14.53 s 2023-09-05 05:36:32,244 44k INFO ====> Epoch: 6434, cost 14.44 s 2023-09-05 05:36:46,862 44k INFO ====> Epoch: 6435, cost 14.62 s 2023-09-05 05:37:01,328 44k INFO ====> Epoch: 6436, cost 14.47 s 2023-09-05 05:37:15,896 44k INFO ====> Epoch: 6437, cost 14.57 s 2023-09-05 05:37:30,285 44k INFO ====> Epoch: 6438, cost 14.39 s 2023-09-05 05:37:31,704 44k INFO Train Epoch: 6439 [5%] 2023-09-05 05:37:31,704 44k INFO Losses: [2.2856271266937256, 2.6534976959228516, 8.018601417541504, 15.448298454284668, 0.6240170001983643], step: 135200, lr: 4.471772184566707e-05, reference_loss: 29.03004264831543 2023-09-05 05:37:37,112 44k INFO Saving model and optimizer state at iteration 6439 to ./logs\44k\G_135200.pth 2023-09-05 05:37:37,770 44k INFO Saving model and optimizer state at iteration 6439 to ./logs\44k\D_135200.pth 2023-09-05 05:37:38,909 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_132800.pth 2023-09-05 05:37:38,946 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_132800.pth 2023-09-05 05:37:52,088 44k INFO ====> Epoch: 6439, cost 21.80 s 2023-09-05 05:38:06,622 44k INFO ====> Epoch: 6440, cost 14.53 s 2023-09-05 05:38:21,172 44k INFO ====> Epoch: 6441, cost 14.55 s 2023-09-05 05:38:35,667 44k INFO ====> Epoch: 6442, cost 14.50 s 2023-09-05 05:38:50,114 44k INFO ====> Epoch: 6443, cost 14.45 s 2023-09-05 05:39:04,532 44k INFO ====> Epoch: 6444, cost 14.42 s 2023-09-05 05:39:18,854 44k INFO ====> Epoch: 6445, cost 14.32 s 2023-09-05 05:39:33,329 44k INFO ====> Epoch: 6446, cost 14.48 s 2023-09-05 05:39:47,966 44k INFO ====> Epoch: 6447, cost 14.64 s 2023-09-05 05:39:57,124 44k INFO Train Epoch: 6448 [57%] 2023-09-05 05:39:57,124 44k INFO Losses: [2.3018033504486084, 2.5587711334228516, 8.753798484802246, 15.412335395812988, 0.5758779048919678], step: 135400, lr: 4.466743955497409e-05, reference_loss: 29.60258674621582 2023-09-05 05:40:02,801 44k INFO ====> Epoch: 6448, cost 14.83 s 2023-09-05 05:40:17,228 44k INFO ====> Epoch: 6449, cost 14.43 s 2023-09-05 05:40:31,671 44k INFO ====> Epoch: 6450, cost 14.44 s 2023-09-05 05:40:46,252 44k INFO ====> Epoch: 6451, cost 14.58 s 2023-09-05 05:41:01,020 44k INFO ====> Epoch: 6452, cost 14.77 s 2023-09-05 05:41:15,392 44k INFO ====> Epoch: 6453, cost 14.37 s 2023-09-05 05:41:29,934 44k INFO ====> Epoch: 6454, cost 14.54 s 2023-09-05 05:41:44,365 44k INFO ====> Epoch: 6455, cost 14.43 s 2023-09-05 05:41:59,081 44k INFO ====> Epoch: 6456, cost 14.72 s 2023-09-05 05:42:13,697 44k INFO ====> Epoch: 6457, cost 14.62 s 2023-09-05 05:42:15,794 44k INFO Train Epoch: 6458 [10%] 2023-09-05 05:42:15,794 44k INFO Losses: [2.4485793113708496, 2.4061453342437744, 6.194052219390869, 14.632217407226562, 0.5618027448654175], step: 135600, lr: 4.461163665185714e-05, reference_loss: 26.242795944213867 2023-09-05 05:42:28,417 44k INFO ====> Epoch: 6458, cost 14.72 s 2023-09-05 05:42:42,902 44k INFO ====> Epoch: 6459, cost 14.48 s 2023-09-05 05:42:57,569 44k INFO ====> Epoch: 6460, cost 14.67 s 2023-09-05 05:43:12,206 44k INFO ====> Epoch: 6461, cost 14.64 s 2023-09-05 05:43:26,592 44k INFO ====> Epoch: 6462, cost 14.39 s 2023-09-05 05:43:41,029 44k INFO ====> Epoch: 6463, cost 14.44 s 2023-09-05 05:43:55,532 44k INFO ====> Epoch: 6464, cost 14.50 s 2023-09-05 05:44:10,312 44k INFO ====> Epoch: 6465, cost 14.78 s 2023-09-05 05:44:24,822 44k INFO ====> Epoch: 6466, cost 14.51 s 2023-09-05 05:44:34,841 44k INFO Train Epoch: 6467 [62%] 2023-09-05 05:44:34,842 44k INFO Losses: [1.9481507539749146, 2.9723589420318604, 9.519732475280762, 14.657835960388184, 0.48452821373939514], step: 135800, lr: 4.456147364735167e-05, reference_loss: 29.58260726928711 2023-09-05 05:44:39,772 44k INFO ====> Epoch: 6467, cost 14.95 s 2023-09-05 05:44:54,311 44k INFO ====> Epoch: 6468, cost 14.54 s 2023-09-05 05:45:09,030 44k INFO ====> Epoch: 6469, cost 14.72 s 2023-09-05 05:45:23,476 44k INFO ====> Epoch: 6470, cost 14.45 s 2023-09-05 05:45:37,784 44k INFO ====> Epoch: 6471, cost 14.31 s 2023-09-05 05:45:52,311 44k INFO ====> Epoch: 6472, cost 14.53 s 2023-09-05 05:46:06,958 44k INFO ====> Epoch: 6473, cost 14.65 s 2023-09-05 05:46:21,462 44k INFO ====> Epoch: 6474, cost 14.50 s 2023-09-05 05:46:35,939 44k INFO ====> Epoch: 6475, cost 14.48 s 2023-09-05 05:46:50,501 44k INFO ====> Epoch: 6476, cost 14.56 s 2023-09-05 05:46:53,358 44k INFO Train Epoch: 6477 [14%] 2023-09-05 05:46:53,358 44k INFO Losses: [2.0154006481170654, 2.942990303039551, 12.09041690826416, 15.044867515563965, 0.6597884297370911], step: 136000, lr: 4.4505803127136804e-05, reference_loss: 32.75346755981445 2023-09-05 05:46:58,831 44k INFO Saving model and optimizer state at iteration 6477 to ./logs\44k\G_136000.pth 2023-09-05 05:46:59,496 44k INFO Saving model and optimizer state at iteration 6477 to ./logs\44k\D_136000.pth 2023-09-05 05:47:00,606 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_133600.pth 2023-09-05 05:47:00,641 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_133600.pth 2023-09-05 05:47:12,454 44k INFO ====> Epoch: 6477, cost 21.95 s 2023-09-05 05:47:27,037 44k INFO ====> Epoch: 6478, cost 14.58 s 2023-09-05 05:47:41,408 44k INFO ====> Epoch: 6479, cost 14.37 s 2023-09-05 05:47:56,317 44k INFO ====> Epoch: 6480, cost 14.91 s 2023-09-05 05:48:10,986 44k INFO ====> Epoch: 6481, cost 14.67 s 2023-09-05 05:48:25,351 44k INFO ====> Epoch: 6482, cost 14.36 s 2023-09-05 05:48:39,945 44k INFO ====> Epoch: 6483, cost 14.59 s 2023-09-05 05:48:54,552 44k INFO ====> Epoch: 6484, cost 14.61 s 2023-09-05 05:49:09,238 44k INFO ====> Epoch: 6485, cost 14.69 s 2023-09-05 05:49:19,637 44k INFO Train Epoch: 6486 [67%] 2023-09-05 05:49:19,637 44k INFO Losses: [2.20234751701355, 2.579890251159668, 9.452425003051758, 15.596482276916504, 0.5386208891868591], step: 136200, lr: 4.4455759125832657e-05, reference_loss: 30.369766235351562 2023-09-05 05:49:23,897 44k INFO ====> Epoch: 6486, cost 14.66 s 2023-09-05 05:49:38,214 44k INFO ====> Epoch: 6487, cost 14.32 s 2023-09-05 05:49:53,076 44k INFO ====> Epoch: 6488, cost 14.86 s 2023-09-05 05:50:07,722 44k INFO ====> Epoch: 6489, cost 14.65 s 2023-09-05 05:50:22,414 44k INFO ====> Epoch: 6490, cost 14.69 s 2023-09-05 05:50:36,936 44k INFO ====> Epoch: 6491, cost 14.52 s 2023-09-05 05:50:51,419 44k INFO ====> Epoch: 6492, cost 14.48 s 2023-09-05 05:51:05,835 44k INFO ====> Epoch: 6493, cost 14.41 s 2023-09-05 05:51:20,206 44k INFO ====> Epoch: 6494, cost 14.37 s 2023-09-05 05:51:34,613 44k INFO ====> Epoch: 6495, cost 14.41 s 2023-09-05 05:51:38,097 44k INFO Train Epoch: 6496 [19%] 2023-09-05 05:51:38,097 44k INFO Losses: [2.1477890014648438, 2.7375831604003906, 10.129276275634766, 15.524454116821289, 0.5264261364936829], step: 136400, lr: 4.4400220674463933e-05, reference_loss: 31.065528869628906 2023-09-05 05:51:49,600 44k INFO ====> Epoch: 6496, cost 14.99 s 2023-09-05 05:52:04,331 44k INFO ====> Epoch: 6497, cost 14.73 s 2023-09-05 05:52:18,846 44k INFO ====> Epoch: 6498, cost 14.52 s 2023-09-05 05:52:33,403 44k INFO ====> Epoch: 6499, cost 14.56 s 2023-09-05 05:52:48,051 44k INFO ====> Epoch: 6500, cost 14.65 s 2023-09-05 05:53:02,710 44k INFO ====> Epoch: 6501, cost 14.66 s 2023-09-05 05:53:17,104 44k INFO ====> Epoch: 6502, cost 14.39 s 2023-09-05 05:53:31,406 44k INFO ====> Epoch: 6503, cost 14.30 s 2023-09-05 05:53:45,944 44k INFO ====> Epoch: 6504, cost 14.54 s 2023-09-05 05:53:57,219 44k INFO Train Epoch: 6505 [71%] 2023-09-05 05:53:57,220 44k INFO Losses: [2.16400408744812, 2.6905128955841064, 10.159835815429688, 14.232665061950684, 0.4570261240005493], step: 136600, lr: 4.4350295394046226e-05, reference_loss: 29.704042434692383 2023-09-05 05:54:00,791 44k INFO ====> Epoch: 6505, cost 14.85 s 2023-09-05 05:54:15,366 44k INFO ====> Epoch: 6506, cost 14.57 s 2023-09-05 05:54:29,799 44k INFO ====> Epoch: 6507, cost 14.43 s 2023-09-05 05:54:44,305 44k INFO ====> Epoch: 6508, cost 14.51 s 2023-09-05 05:54:59,067 44k INFO ====> Epoch: 6509, cost 14.76 s 2023-09-05 05:55:13,684 44k INFO ====> Epoch: 6510, cost 14.62 s 2023-09-05 05:55:28,019 44k INFO ====> Epoch: 6511, cost 14.34 s 2023-09-05 05:55:42,484 44k INFO ====> Epoch: 6512, cost 14.46 s 2023-09-05 05:55:57,228 44k INFO ====> Epoch: 6513, cost 14.74 s 2023-09-05 05:56:11,895 44k INFO ====> Epoch: 6514, cost 14.67 s 2023-09-05 05:56:16,231 44k INFO Train Epoch: 6515 [24%] 2023-09-05 05:56:16,232 44k INFO Losses: [2.508110761642456, 2.171219825744629, 7.418562412261963, 13.863261222839355, 0.676226794719696], step: 136800, lr: 4.429488869821276e-05, reference_loss: 26.63738250732422 2023-09-05 05:56:21,653 44k INFO Saving model and optimizer state at iteration 6515 to ./logs\44k\G_136800.pth 2023-09-05 05:56:22,233 44k INFO Saving model and optimizer state at iteration 6515 to ./logs\44k\D_136800.pth 2023-09-05 05:56:23,517 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_134400.pth 2023-09-05 05:56:23,553 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_134400.pth 2023-09-05 05:56:33,908 44k INFO ====> Epoch: 6515, cost 22.01 s 2023-09-05 05:56:48,462 44k INFO ====> Epoch: 6516, cost 14.55 s 2023-09-05 05:57:02,930 44k INFO ====> Epoch: 6517, cost 14.47 s 2023-09-05 05:57:17,274 44k INFO ====> Epoch: 6518, cost 14.34 s 2023-09-05 05:57:31,730 44k INFO ====> Epoch: 6519, cost 14.46 s 2023-09-05 05:57:46,313 44k INFO ====> Epoch: 6520, cost 14.58 s 2023-09-05 05:58:00,739 44k INFO ====> Epoch: 6521, cost 14.43 s 2023-09-05 05:58:15,319 44k INFO ====> Epoch: 6522, cost 14.58 s 2023-09-05 05:58:29,985 44k INFO ====> Epoch: 6523, cost 14.67 s 2023-09-05 05:58:42,034 44k INFO Train Epoch: 6524 [76%] 2023-09-05 05:58:42,035 44k INFO Losses: [2.268833637237549, 2.436344623565674, 10.941821098327637, 16.77876853942871, 0.7476362586021423], step: 137000, lr: 4.424508185703638e-05, reference_loss: 33.17340087890625 2023-09-05 05:58:44,783 44k INFO ====> Epoch: 6524, cost 14.80 s 2023-09-05 05:58:59,370 44k INFO ====> Epoch: 6525, cost 14.59 s 2023-09-05 05:59:13,989 44k INFO ====> Epoch: 6526, cost 14.62 s 2023-09-05 05:59:28,306 44k INFO ====> Epoch: 6527, cost 14.32 s 2023-09-05 05:59:42,831 44k INFO ====> Epoch: 6528, cost 14.53 s 2023-09-05 05:59:57,470 44k INFO ====> Epoch: 6529, cost 14.64 s 2023-09-05 06:00:12,690 44k INFO ====> Epoch: 6530, cost 15.22 s 2023-09-05 06:00:27,357 44k INFO ====> Epoch: 6531, cost 14.67 s 2023-09-05 06:00:41,952 44k INFO ====> Epoch: 6532, cost 14.59 s 2023-09-05 06:00:56,532 44k INFO ====> Epoch: 6533, cost 14.58 s 2023-09-05 06:01:01,446 44k INFO Train Epoch: 6534 [29%] 2023-09-05 06:01:01,446 44k INFO Losses: [2.3524136543273926, 2.5908145904541016, 9.094012260437012, 15.545509338378906, 0.522716224193573], step: 137200, lr: 4.4189806604170574e-05, reference_loss: 30.105466842651367 2023-09-05 06:01:11,412 44k INFO ====> Epoch: 6534, cost 14.88 s 2023-09-05 06:01:25,815 44k INFO ====> Epoch: 6535, cost 14.40 s 2023-09-05 06:01:40,193 44k INFO ====> Epoch: 6536, cost 14.38 s 2023-09-05 06:01:54,760 44k INFO ====> Epoch: 6537, cost 14.57 s 2023-09-05 06:02:09,419 44k INFO ====> Epoch: 6538, cost 14.66 s 2023-09-05 06:02:23,834 44k INFO ====> Epoch: 6539, cost 14.41 s 2023-09-05 06:02:38,337 44k INFO ====> Epoch: 6540, cost 14.50 s 2023-09-05 06:02:54,332 44k INFO ====> Epoch: 6541, cost 15.99 s 2023-09-05 06:03:09,563 44k INFO ====> Epoch: 6542, cost 15.23 s 2023-09-05 06:03:22,326 44k INFO Train Epoch: 6543 [81%] 2023-09-05 06:03:22,326 44k INFO Losses: [2.2772302627563477, 2.6574602127075195, 7.617188930511475, 14.2283353805542, 0.7393026351928711], step: 137400, lr: 4.4140117921258554e-05, reference_loss: 27.519519805908203 2023-09-05 06:03:24,491 44k INFO ====> Epoch: 6543, cost 14.93 s 2023-09-05 06:03:39,230 44k INFO ====> Epoch: 6544, cost 14.74 s 2023-09-05 06:03:54,013 44k INFO ====> Epoch: 6545, cost 14.78 s 2023-09-05 06:04:08,772 44k INFO ====> Epoch: 6546, cost 14.76 s 2023-09-05 06:04:23,201 44k INFO ====> Epoch: 6547, cost 14.43 s 2023-09-05 06:04:37,877 44k INFO ====> Epoch: 6548, cost 14.68 s 2023-09-05 06:04:52,805 44k INFO ====> Epoch: 6549, cost 14.93 s 2023-09-05 06:05:07,439 44k INFO ====> Epoch: 6550, cost 14.64 s 2023-09-05 06:05:21,952 44k INFO ====> Epoch: 6551, cost 14.51 s 2023-09-05 06:05:36,351 44k INFO ====> Epoch: 6552, cost 14.40 s 2023-09-05 06:05:42,084 44k INFO Train Epoch: 6553 [33%] 2023-09-05 06:05:42,085 44k INFO Losses: [1.961801528930664, 2.8738503456115723, 8.540290832519531, 15.759729385375977, 0.6091558933258057], step: 137600, lr: 4.4084973799534296e-05, reference_loss: 29.744827270507812 2023-09-05 06:05:47,765 44k INFO Saving model and optimizer state at iteration 6553 to ./logs\44k\G_137600.pth 2023-09-05 06:05:48,378 44k INFO Saving model and optimizer state at iteration 6553 to ./logs\44k\D_137600.pth 2023-09-05 06:05:49,594 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_135200.pth 2023-09-05 06:05:49,636 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_135200.pth 2023-09-05 06:05:58,630 44k INFO ====> Epoch: 6553, cost 22.28 s 2023-09-05 06:06:13,309 44k INFO ====> Epoch: 6554, cost 14.68 s 2023-09-05 06:06:27,947 44k INFO ====> Epoch: 6555, cost 14.64 s 2023-09-05 06:06:42,408 44k INFO ====> Epoch: 6556, cost 14.46 s 2023-09-05 06:06:57,008 44k INFO ====> Epoch: 6557, cost 14.60 s 2023-09-05 06:07:11,652 44k INFO ====> Epoch: 6558, cost 14.64 s 2023-09-05 06:07:26,039 44k INFO ====> Epoch: 6559, cost 14.39 s 2023-09-05 06:07:40,634 44k INFO ====> Epoch: 6560, cost 14.60 s 2023-09-05 06:07:55,346 44k INFO ====> Epoch: 6561, cost 14.71 s 2023-09-05 06:08:08,733 44k INFO Train Epoch: 6562 [86%] 2023-09-05 06:08:08,733 44k INFO Losses: [2.0284953117370605, 2.9968810081481934, 9.710783958435059, 14.937870025634766, 0.6904305815696716], step: 137800, lr: 4.403540299457623e-05, reference_loss: 30.364459991455078 2023-09-05 06:08:10,253 44k INFO ====> Epoch: 6562, cost 14.91 s 2023-09-05 06:08:24,807 44k INFO ====> Epoch: 6563, cost 14.55 s 2023-09-05 06:08:39,538 44k INFO ====> Epoch: 6564, cost 14.73 s 2023-09-05 06:08:54,108 44k INFO ====> Epoch: 6565, cost 14.57 s 2023-09-05 06:09:08,790 44k INFO ====> Epoch: 6566, cost 14.68 s 2023-09-05 06:09:23,578 44k INFO ====> Epoch: 6567, cost 14.79 s 2023-09-05 06:09:38,179 44k INFO ====> Epoch: 6568, cost 14.60 s 2023-09-05 06:09:52,879 44k INFO ====> Epoch: 6569, cost 14.70 s 2023-09-05 06:10:07,811 44k INFO ====> Epoch: 6570, cost 14.93 s 2023-09-05 06:10:22,451 44k INFO ====> Epoch: 6571, cost 14.64 s 2023-09-05 06:10:28,900 44k INFO Train Epoch: 6572 [38%] 2023-09-05 06:10:28,901 44k INFO Losses: [2.3206701278686523, 2.3830857276916504, 8.871476173400879, 15.587055206298828, 0.5364035964012146], step: 138000, lr: 4.398038969290718e-05, reference_loss: 29.69869041442871 2023-09-05 06:10:37,409 44k INFO ====> Epoch: 6572, cost 14.96 s 2023-09-05 06:10:52,213 44k INFO ====> Epoch: 6573, cost 14.80 s 2023-09-05 06:11:06,942 44k INFO ====> Epoch: 6574, cost 14.73 s 2023-09-05 06:11:21,616 44k INFO ====> Epoch: 6575, cost 14.67 s 2023-09-05 06:11:36,268 44k INFO ====> Epoch: 6576, cost 14.65 s 2023-09-05 06:11:51,028 44k INFO ====> Epoch: 6577, cost 14.76 s 2023-09-05 06:12:05,750 44k INFO ====> Epoch: 6578, cost 14.72 s 2023-09-05 06:12:20,472 44k INFO ====> Epoch: 6579, cost 14.72 s 2023-09-05 06:12:34,949 44k INFO ====> Epoch: 6580, cost 14.48 s 2023-09-05 06:12:49,155 44k INFO Train Epoch: 6581 [90%] 2023-09-05 06:12:49,155 44k INFO Losses: [2.238065004348755, 2.7079412937164307, 8.90467357635498, 16.630958557128906, 0.3996330499649048], step: 138200, lr: 4.393093648625767e-05, reference_loss: 30.881271362304688 2023-09-05 06:12:49,855 44k INFO ====> Epoch: 6581, cost 14.91 s 2023-09-05 06:13:04,712 44k INFO ====> Epoch: 6582, cost 14.86 s 2023-09-05 06:13:19,222 44k INFO ====> Epoch: 6583, cost 14.51 s 2023-09-05 06:13:33,552 44k INFO ====> Epoch: 6584, cost 14.33 s 2023-09-05 06:13:48,131 44k INFO ====> Epoch: 6585, cost 14.58 s 2023-09-05 06:14:02,773 44k INFO ====> Epoch: 6586, cost 14.64 s 2023-09-05 06:14:17,475 44k INFO ====> Epoch: 6587, cost 14.70 s 2023-09-05 06:14:31,796 44k INFO ====> Epoch: 6588, cost 14.32 s 2023-09-05 06:14:46,284 44k INFO ====> Epoch: 6589, cost 14.49 s 2023-09-05 06:15:00,650 44k INFO ====> Epoch: 6590, cost 14.37 s 2023-09-05 06:15:07,735 44k INFO Train Epoch: 6591 [43%] 2023-09-05 06:15:07,735 44k INFO Losses: [2.1399288177490234, 2.955252170562744, 8.149455070495605, 14.69959831237793, 0.6973069310188293], step: 138400, lr: 4.387605369429548e-05, reference_loss: 28.641542434692383 2023-09-05 06:15:13,382 44k INFO Saving model and optimizer state at iteration 6591 to ./logs\44k\G_138400.pth 2023-09-05 06:15:14,002 44k INFO Saving model and optimizer state at iteration 6591 to ./logs\44k\D_138400.pth 2023-09-05 06:15:15,184 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_136000.pth 2023-09-05 06:15:15,219 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_136000.pth 2023-09-05 06:15:22,706 44k INFO ====> Epoch: 6591, cost 22.06 s 2023-09-05 06:15:37,210 44k INFO ====> Epoch: 6592, cost 14.50 s 2023-09-05 06:15:51,634 44k INFO ====> Epoch: 6593, cost 14.42 s 2023-09-05 06:16:06,192 44k INFO ====> Epoch: 6594, cost 14.56 s 2023-09-05 06:16:20,592 44k INFO ====> Epoch: 6595, cost 14.40 s 2023-09-05 06:16:35,113 44k INFO ====> Epoch: 6596, cost 14.52 s 2023-09-05 06:16:49,658 44k INFO ====> Epoch: 6597, cost 14.55 s 2023-09-05 06:17:04,345 44k INFO ====> Epoch: 6598, cost 14.69 s 2023-09-05 06:17:18,915 44k INFO ====> Epoch: 6599, cost 14.57 s 2023-09-05 06:17:33,277 44k INFO Train Epoch: 6600 [95%] 2023-09-05 06:17:33,277 44k INFO Losses: [1.8116514682769775, 3.39630126953125, 9.54498291015625, 14.347105026245117, 0.5404459238052368], step: 138600, lr: 4.3826717806972515e-05, reference_loss: 29.640485763549805 2023-09-05 06:17:33,539 44k INFO ====> Epoch: 6600, cost 14.62 s 2023-09-05 06:17:48,242 44k INFO ====> Epoch: 6601, cost 14.70 s 2023-09-05 06:18:02,853 44k INFO ====> Epoch: 6602, cost 14.61 s 2023-09-05 06:18:17,426 44k INFO ====> Epoch: 6603, cost 14.57 s 2023-09-05 06:18:31,807 44k INFO ====> Epoch: 6604, cost 14.38 s 2023-09-05 06:18:46,409 44k INFO ====> Epoch: 6605, cost 14.60 s 2023-09-05 06:19:01,195 44k INFO ====> Epoch: 6606, cost 14.79 s 2023-09-05 06:19:15,768 44k INFO ====> Epoch: 6607, cost 14.57 s 2023-09-05 06:19:30,140 44k INFO ====> Epoch: 6608, cost 14.37 s 2023-09-05 06:19:44,480 44k INFO ====> Epoch: 6609, cost 14.34 s 2023-09-05 06:19:52,371 44k INFO Train Epoch: 6610 [48%] 2023-09-05 06:19:52,372 44k INFO Losses: [2.1895432472229004, 2.730229377746582, 12.570185661315918, 17.343923568725586, 0.6376988291740417], step: 138800, lr: 4.377196521510509e-05, reference_loss: 35.471580505371094 2023-09-05 06:19:59,482 44k INFO ====> Epoch: 6610, cost 15.00 s 2023-09-05 06:20:14,017 44k INFO ====> Epoch: 6611, cost 14.54 s 2023-09-05 06:20:28,565 44k INFO ====> Epoch: 6612, cost 14.55 s 2023-09-05 06:20:43,161 44k INFO ====> Epoch: 6613, cost 14.60 s 2023-09-05 06:20:57,552 44k INFO ====> Epoch: 6614, cost 14.39 s 2023-09-05 06:21:12,016 44k INFO ====> Epoch: 6615, cost 14.46 s 2023-09-05 06:21:26,463 44k INFO ====> Epoch: 6616, cost 14.45 s 2023-09-05 06:21:40,945 44k INFO ====> Epoch: 6617, cost 14.48 s 2023-09-05 06:21:55,610 44k INFO ====> Epoch: 6618, cost 14.66 s 2023-09-05 06:22:10,132 44k INFO ====> Epoch: 6619, cost 14.52 s 2023-09-05 06:22:10,845 44k INFO Train Epoch: 6620 [0%] 2023-09-05 06:22:10,845 44k INFO Losses: [2.1099905967712402, 2.7917680740356445, 8.986787796020508, 15.15225601196289, 0.6510506272315979], step: 139000, lr: 4.371728102549242e-05, reference_loss: 29.691852569580078 2023-09-05 06:22:24,856 44k INFO ====> Epoch: 6620, cost 14.72 s 2023-09-05 06:22:39,408 44k INFO ====> Epoch: 6621, cost 14.55 s 2023-09-05 06:22:53,944 44k INFO ====> Epoch: 6622, cost 14.54 s 2023-09-05 06:23:08,586 44k INFO ====> Epoch: 6623, cost 14.64 s 2023-09-05 06:23:23,034 44k INFO ====> Epoch: 6624, cost 14.45 s 2023-09-05 06:23:37,427 44k INFO ====> Epoch: 6625, cost 14.39 s 2023-09-05 06:23:52,139 44k INFO ====> Epoch: 6626, cost 14.71 s 2023-09-05 06:24:07,107 44k INFO ====> Epoch: 6627, cost 14.97 s 2023-09-05 06:24:21,500 44k INFO ====> Epoch: 6628, cost 14.39 s 2023-09-05 06:24:29,769 44k INFO Train Epoch: 6629 [52%] 2023-09-05 06:24:29,770 44k INFO Losses: [2.489692211151123, 2.3550515174865723, 8.798820495605469, 14.961256980895996, 0.6540064215660095], step: 139200, lr: 4.366812366813828e-05, reference_loss: 29.258827209472656 2023-09-05 06:24:35,375 44k INFO Saving model and optimizer state at iteration 6629 to ./logs\44k\G_139200.pth 2023-09-05 06:24:35,926 44k INFO Saving model and optimizer state at iteration 6629 to ./logs\44k\D_139200.pth 2023-09-05 06:24:37,059 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_136800.pth 2023-09-05 06:24:37,094 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_136800.pth 2023-09-05 06:24:43,152 44k INFO ====> Epoch: 6629, cost 21.65 s 2023-09-05 06:24:57,698 44k INFO ====> Epoch: 6630, cost 14.55 s 2023-09-05 06:25:12,203 44k INFO ====> Epoch: 6631, cost 14.50 s 2023-09-05 06:25:26,490 44k INFO ====> Epoch: 6632, cost 14.29 s 2023-09-05 06:25:40,763 44k INFO ====> Epoch: 6633, cost 14.27 s 2023-09-05 06:25:55,335 44k INFO ====> Epoch: 6634, cost 14.57 s 2023-09-05 06:26:09,961 44k INFO ====> Epoch: 6635, cost 14.63 s 2023-09-05 06:26:24,415 44k INFO ====> Epoch: 6636, cost 14.45 s 2023-09-05 06:26:39,073 44k INFO ====> Epoch: 6637, cost 14.66 s 2023-09-05 06:26:53,588 44k INFO ====> Epoch: 6638, cost 14.51 s 2023-09-05 06:26:55,069 44k INFO Train Epoch: 6639 [5%] 2023-09-05 06:26:55,070 44k INFO Losses: [1.765135645866394, 3.15523099899292, 9.155001640319824, 13.11817741394043, 0.6118858456611633], step: 139400, lr: 4.361356920747007e-05, reference_loss: 27.805431365966797 2023-09-05 06:27:08,575 44k INFO ====> Epoch: 6639, cost 14.99 s 2023-09-05 06:27:22,908 44k INFO ====> Epoch: 6640, cost 14.33 s 2023-09-05 06:27:37,242 44k INFO ====> Epoch: 6641, cost 14.33 s 2023-09-05 06:27:51,777 44k INFO ====> Epoch: 6642, cost 14.54 s 2023-09-05 06:28:06,361 44k INFO ====> Epoch: 6643, cost 14.58 s 2023-09-05 06:28:20,972 44k INFO ====> Epoch: 6644, cost 14.61 s 2023-09-05 06:28:35,761 44k INFO ====> Epoch: 6645, cost 14.79 s 2023-09-05 06:28:50,257 44k INFO ====> Epoch: 6646, cost 14.50 s 2023-09-05 06:29:04,863 44k INFO ====> Epoch: 6647, cost 14.61 s 2023-09-05 06:29:14,109 44k INFO Train Epoch: 6648 [57%] 2023-09-05 06:29:14,109 44k INFO Losses: [2.3719258308410645, 2.5445141792297363, 10.931586265563965, 16.5211181640625, 0.5121169686317444], step: 139600, lr: 4.3564528467590325e-05, reference_loss: 32.88125991821289 2023-09-05 06:29:19,704 44k INFO ====> Epoch: 6648, cost 14.84 s 2023-09-05 06:29:34,187 44k INFO ====> Epoch: 6649, cost 14.48 s 2023-09-05 06:29:48,788 44k INFO ====> Epoch: 6650, cost 14.60 s 2023-09-05 06:30:03,568 44k INFO ====> Epoch: 6651, cost 14.78 s 2023-09-05 06:30:18,276 44k INFO ====> Epoch: 6652, cost 14.71 s 2023-09-05 06:30:32,761 44k INFO ====> Epoch: 6653, cost 14.48 s 2023-09-05 06:30:47,249 44k INFO ====> Epoch: 6654, cost 14.49 s 2023-09-05 06:31:01,865 44k INFO ====> Epoch: 6655, cost 14.62 s 2023-09-05 06:31:16,319 44k INFO ====> Epoch: 6656, cost 14.45 s 2023-09-05 06:31:30,799 44k INFO ====> Epoch: 6657, cost 14.48 s 2023-09-05 06:31:32,957 44k INFO Train Epoch: 6658 [10%] 2023-09-05 06:31:32,957 44k INFO Losses: [2.03568959236145, 3.271174192428589, 9.927366256713867, 14.806466102600098, 0.631647527217865], step: 139800, lr: 4.3510103428106684e-05, reference_loss: 30.67234230041504 2023-09-05 06:31:45,583 44k INFO ====> Epoch: 6658, cost 14.78 s 2023-09-05 06:32:00,193 44k INFO ====> Epoch: 6659, cost 14.61 s 2023-09-05 06:32:14,870 44k INFO ====> Epoch: 6660, cost 14.68 s 2023-09-05 06:32:29,412 44k INFO ====> Epoch: 6661, cost 14.54 s 2023-09-05 06:32:43,952 44k INFO ====> Epoch: 6662, cost 14.54 s 2023-09-05 06:32:58,574 44k INFO ====> Epoch: 6663, cost 14.62 s 2023-09-05 06:33:13,118 44k INFO ====> Epoch: 6664, cost 14.54 s 2023-09-05 06:33:27,479 44k INFO ====> Epoch: 6665, cost 14.36 s 2023-09-05 06:33:41,840 44k INFO ====> Epoch: 6666, cost 14.36 s 2023-09-05 06:33:51,767 44k INFO Train Epoch: 6667 [62%] 2023-09-05 06:33:51,768 44k INFO Losses: [2.2767739295959473, 2.4349048137664795, 8.214366912841797, 15.316532135009766, 0.5639501214027405], step: 140000, lr: 4.3461179029046195e-05, reference_loss: 28.80652618408203 2023-09-05 06:33:57,262 44k INFO Saving model and optimizer state at iteration 6667 to ./logs\44k\G_140000.pth 2023-09-05 06:33:57,927 44k INFO Saving model and optimizer state at iteration 6667 to ./logs\44k\D_140000.pth 2023-09-05 06:33:59,039 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_137600.pth 2023-09-05 06:33:59,074 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_137600.pth 2023-09-05 06:34:03,760 44k INFO ====> Epoch: 6667, cost 21.92 s 2023-09-05 06:34:18,187 44k INFO ====> Epoch: 6668, cost 14.43 s 2023-09-05 06:34:32,659 44k INFO ====> Epoch: 6669, cost 14.47 s 2023-09-05 06:34:47,288 44k INFO ====> Epoch: 6670, cost 14.63 s 2023-09-05 06:35:02,004 44k INFO ====> Epoch: 6671, cost 14.72 s 2023-09-05 06:35:16,569 44k INFO ====> Epoch: 6672, cost 14.56 s 2023-09-05 06:35:30,837 44k INFO ====> Epoch: 6673, cost 14.27 s 2023-09-05 06:35:45,223 44k INFO ====> Epoch: 6674, cost 14.39 s 2023-09-05 06:36:00,142 44k INFO ====> Epoch: 6675, cost 14.92 s 2023-09-05 06:36:14,958 44k INFO ====> Epoch: 6676, cost 14.82 s 2023-09-05 06:36:17,869 44k INFO Train Epoch: 6677 [14%] 2023-09-05 06:36:17,869 44k INFO Losses: [2.1426773071289062, 2.81526255607605, 8.153197288513184, 16.565853118896484, 0.7036107778549194], step: 140200, lr: 4.340688310371738e-05, reference_loss: 30.38060188293457 2023-09-05 06:36:29,936 44k INFO ====> Epoch: 6677, cost 14.98 s 2023-09-05 06:36:44,625 44k INFO ====> Epoch: 6678, cost 14.69 s 2023-09-05 06:36:59,187 44k INFO ====> Epoch: 6679, cost 14.56 s 2023-09-05 06:37:13,605 44k INFO ====> Epoch: 6680, cost 14.42 s 2023-09-05 06:37:27,954 44k INFO ====> Epoch: 6681, cost 14.35 s 2023-09-05 06:37:42,470 44k INFO ====> Epoch: 6682, cost 14.52 s 2023-09-05 06:37:57,302 44k INFO ====> Epoch: 6683, cost 14.83 s 2023-09-05 06:38:12,058 44k INFO ====> Epoch: 6684, cost 14.76 s 2023-09-05 06:38:26,624 44k INFO ====> Epoch: 6685, cost 14.57 s 2023-09-05 06:38:37,191 44k INFO Train Epoch: 6686 [67%] 2023-09-05 06:38:37,191 44k INFO Losses: [2.1606273651123047, 2.728667736053467, 11.356027603149414, 16.182336807250977, 0.5489596128463745], step: 140400, lr: 4.3358074769477326e-05, reference_loss: 32.97661590576172 2023-09-05 06:38:41,386 44k INFO ====> Epoch: 6686, cost 14.76 s 2023-09-05 06:38:56,005 44k INFO ====> Epoch: 6687, cost 14.62 s 2023-09-05 06:39:10,483 44k INFO ====> Epoch: 6688, cost 14.48 s 2023-09-05 06:39:24,693 44k INFO ====> Epoch: 6689, cost 14.21 s 2023-09-05 06:39:39,084 44k INFO ====> Epoch: 6690, cost 14.39 s 2023-09-05 06:39:53,672 44k INFO ====> Epoch: 6691, cost 14.59 s 2023-09-05 06:40:08,215 44k INFO ====> Epoch: 6692, cost 14.54 s 2023-09-05 06:40:22,771 44k INFO ====> Epoch: 6693, cost 14.56 s 2023-09-05 06:40:37,307 44k INFO ====> Epoch: 6694, cost 14.54 s 2023-09-05 06:40:52,034 44k INFO ====> Epoch: 6695, cost 14.73 s 2023-09-05 06:40:55,557 44k INFO Train Epoch: 6696 [19%] 2023-09-05 06:40:55,557 44k INFO Losses: [2.5046067237854004, 2.1620945930480957, 9.008230209350586, 15.084769248962402, 0.492430180311203], step: 140600, lr: 4.3303907652001954e-05, reference_loss: 29.25213050842285 2023-09-05 06:41:06,775 44k INFO ====> Epoch: 6696, cost 14.74 s 2023-09-05 06:41:21,183 44k INFO ====> Epoch: 6697, cost 14.41 s 2023-09-05 06:41:35,592 44k INFO ====> Epoch: 6698, cost 14.41 s 2023-09-05 06:41:50,334 44k INFO ====> Epoch: 6699, cost 14.74 s 2023-09-05 06:42:05,047 44k INFO ====> Epoch: 6700, cost 14.71 s 2023-09-05 06:42:19,700 44k INFO ====> Epoch: 6701, cost 14.65 s 2023-09-05 06:42:34,190 44k INFO ====> Epoch: 6702, cost 14.49 s 2023-09-05 06:42:48,646 44k INFO ====> Epoch: 6703, cost 14.46 s 2023-09-05 06:43:03,099 44k INFO ====> Epoch: 6704, cost 14.45 s 2023-09-05 06:43:14,454 44k INFO Train Epoch: 6705 [71%] 2023-09-05 06:43:14,455 44k INFO Losses: [2.202936887741089, 2.6270766258239746, 8.191397666931152, 15.04189682006836, 0.6239835023880005], step: 140800, lr: 4.325521510723828e-05, reference_loss: 28.687292098999023 2023-09-05 06:43:19,868 44k INFO Saving model and optimizer state at iteration 6705 to ./logs\44k\G_140800.pth 2023-09-05 06:43:20,461 44k INFO Saving model and optimizer state at iteration 6705 to ./logs\44k\D_140800.pth 2023-09-05 06:43:21,567 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_138400.pth 2023-09-05 06:43:21,604 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_138400.pth 2023-09-05 06:43:24,837 44k INFO ====> Epoch: 6705, cost 21.74 s 2023-09-05 06:43:39,237 44k INFO ====> Epoch: 6706, cost 14.40 s 2023-09-05 06:43:54,018 44k INFO ====> Epoch: 6707, cost 14.78 s 2023-09-05 06:44:08,862 44k INFO ====> Epoch: 6708, cost 14.84 s 2023-09-05 06:44:23,436 44k INFO ====> Epoch: 6709, cost 14.57 s 2023-09-05 06:44:38,051 44k INFO ====> Epoch: 6710, cost 14.61 s 2023-09-05 06:44:52,526 44k INFO ====> Epoch: 6711, cost 14.48 s 2023-09-05 06:45:07,094 44k INFO ====> Epoch: 6712, cost 14.57 s 2023-09-05 06:45:21,636 44k INFO ====> Epoch: 6713, cost 14.54 s 2023-09-05 06:45:35,960 44k INFO ====> Epoch: 6714, cost 14.32 s 2023-09-05 06:45:40,153 44k INFO Train Epoch: 6715 [24%] 2023-09-05 06:45:40,154 44k INFO Losses: [2.347276210784912, 2.4009010791778564, 9.294133186340332, 14.885658264160156, 0.6289687752723694], step: 141000, lr: 4.3201176492041625e-05, reference_loss: 29.55693817138672 2023-09-05 06:45:50,625 44k INFO ====> Epoch: 6715, cost 14.67 s 2023-09-05 06:46:05,291 44k INFO ====> Epoch: 6716, cost 14.67 s 2023-09-05 06:46:19,985 44k INFO ====> Epoch: 6717, cost 14.69 s 2023-09-05 06:46:34,426 44k INFO ====> Epoch: 6718, cost 14.44 s 2023-09-05 06:46:49,022 44k INFO ====> Epoch: 6719, cost 14.60 s 2023-09-05 06:47:03,561 44k INFO ====> Epoch: 6720, cost 14.54 s 2023-09-05 06:47:17,962 44k INFO ====> Epoch: 6721, cost 14.40 s 2023-09-05 06:47:32,288 44k INFO ====> Epoch: 6722, cost 14.33 s 2023-09-05 06:47:47,045 44k INFO ====> Epoch: 6723, cost 14.76 s 2023-09-05 06:47:59,104 44k INFO Train Epoch: 6724 [76%] 2023-09-05 06:47:59,105 44k INFO Losses: [2.2035865783691406, 2.871272087097168, 10.122400283813477, 16.633533477783203, 0.5336818695068359], step: 141200, lr: 4.315259946206348e-05, reference_loss: 32.364471435546875 2023-09-05 06:48:01,891 44k INFO ====> Epoch: 6724, cost 14.85 s 2023-09-05 06:48:16,523 44k INFO ====> Epoch: 6725, cost 14.63 s 2023-09-05 06:48:31,102 44k INFO ====> Epoch: 6726, cost 14.58 s 2023-09-05 06:48:45,719 44k INFO ====> Epoch: 6727, cost 14.62 s 2023-09-05 06:49:00,328 44k INFO ====> Epoch: 6728, cost 14.61 s 2023-09-05 06:49:14,726 44k INFO ====> Epoch: 6729, cost 14.40 s 2023-09-05 06:49:29,230 44k INFO ====> Epoch: 6730, cost 14.50 s 2023-09-05 06:49:43,627 44k INFO ====> Epoch: 6731, cost 14.40 s 2023-09-05 06:49:58,343 44k INFO ====> Epoch: 6732, cost 14.72 s 2023-09-05 06:50:12,942 44k INFO ====> Epoch: 6733, cost 14.60 s 2023-09-05 06:50:17,869 44k INFO Train Epoch: 6734 [29%] 2023-09-05 06:50:17,869 44k INFO Losses: [2.445235252380371, 2.3541605472564697, 8.31765079498291, 13.914798736572266, 0.4644429683685303], step: 141400, lr: 4.3098689044295694e-05, reference_loss: 27.496288299560547 2023-09-05 06:50:27,616 44k INFO ====> Epoch: 6734, cost 14.67 s 2023-09-05 06:50:42,264 44k INFO ====> Epoch: 6735, cost 14.65 s 2023-09-05 06:50:56,801 44k INFO ====> Epoch: 6736, cost 14.54 s 2023-09-05 06:51:11,314 44k INFO ====> Epoch: 6737, cost 14.51 s 2023-09-05 06:51:25,706 44k INFO ====> Epoch: 6738, cost 14.39 s 2023-09-05 06:51:40,253 44k INFO ====> Epoch: 6739, cost 14.55 s 2023-09-05 06:51:54,733 44k INFO ====> Epoch: 6740, cost 14.48 s 2023-09-05 06:52:09,403 44k INFO ====> Epoch: 6741, cost 14.67 s 2023-09-05 06:52:23,872 44k INFO ====> Epoch: 6742, cost 14.47 s 2023-09-05 06:52:36,480 44k INFO Train Epoch: 6743 [81%] 2023-09-05 06:52:36,480 44k INFO Losses: [2.450932741165161, 2.3206686973571777, 8.753357887268066, 14.21786117553711, 0.5666757225990295], step: 141600, lr: 4.305022725506388e-05, reference_loss: 28.30949592590332 2023-09-05 06:52:41,927 44k INFO Saving model and optimizer state at iteration 6743 to ./logs\44k\G_141600.pth 2023-09-05 06:52:42,509 44k INFO Saving model and optimizer state at iteration 6743 to ./logs\44k\D_141600.pth 2023-09-05 06:52:43,762 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_139200.pth 2023-09-05 06:52:43,797 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_139200.pth 2023-09-05 06:52:45,685 44k INFO ====> Epoch: 6743, cost 21.81 s 2023-09-05 06:53:00,354 44k INFO ====> Epoch: 6744, cost 14.67 s 2023-09-05 06:53:15,015 44k INFO ====> Epoch: 6745, cost 14.66 s 2023-09-05 06:53:29,403 44k INFO ====> Epoch: 6746, cost 14.39 s 2023-09-05 06:53:43,949 44k INFO ====> Epoch: 6747, cost 14.55 s 2023-09-05 06:53:58,731 44k INFO ====> Epoch: 6748, cost 14.78 s 2023-09-05 06:54:13,298 44k INFO ====> Epoch: 6749, cost 14.57 s 2023-09-05 06:54:27,735 44k INFO ====> Epoch: 6750, cost 14.44 s 2023-09-05 06:54:42,230 44k INFO ====> Epoch: 6751, cost 14.50 s 2023-09-05 06:54:56,861 44k INFO ====> Epoch: 6752, cost 14.63 s 2023-09-05 06:55:02,469 44k INFO Train Epoch: 6753 [33%] 2023-09-05 06:55:02,469 44k INFO Losses: [2.091369152069092, 2.742460250854492, 8.918657302856445, 14.81006908416748, 0.6116328239440918], step: 141800, lr: 4.299644473059838e-05, reference_loss: 29.174190521240234 2023-09-05 06:55:11,710 44k INFO ====> Epoch: 6753, cost 14.85 s 2023-09-05 06:55:26,078 44k INFO ====> Epoch: 6754, cost 14.37 s 2023-09-05 06:55:40,469 44k INFO ====> Epoch: 6755, cost 14.39 s 2023-09-05 06:55:55,062 44k INFO ====> Epoch: 6756, cost 14.59 s 2023-09-05 06:56:09,644 44k INFO ====> Epoch: 6757, cost 14.58 s 2023-09-05 06:56:24,171 44k INFO ====> Epoch: 6758, cost 14.53 s 2023-09-05 06:56:38,395 44k INFO ====> Epoch: 6759, cost 14.22 s 2023-09-05 06:56:53,442 44k INFO ====> Epoch: 6760, cost 15.05 s 2023-09-05 06:57:08,553 44k INFO ====> Epoch: 6761, cost 15.11 s 2023-09-05 06:57:22,215 44k INFO Train Epoch: 6762 [86%] 2023-09-05 06:57:22,215 44k INFO Losses: [2.6345086097717285, 2.2948861122131348, 6.7702765464782715, 14.620979309082031, 0.5063795447349548], step: 142000, lr: 4.2948097908723815e-05, reference_loss: 26.827030181884766 2023-09-05 06:57:23,720 44k INFO ====> Epoch: 6762, cost 15.17 s 2023-09-05 06:57:38,276 44k INFO ====> Epoch: 6763, cost 14.56 s 2023-09-05 06:57:52,949 44k INFO ====> Epoch: 6764, cost 14.67 s 2023-09-05 06:58:07,612 44k INFO ====> Epoch: 6765, cost 14.66 s 2023-09-05 06:58:22,229 44k INFO ====> Epoch: 6766, cost 14.62 s 2023-09-05 06:58:36,844 44k INFO ====> Epoch: 6767, cost 14.61 s 2023-09-05 06:58:51,666 44k INFO ====> Epoch: 6768, cost 14.82 s 2023-09-05 06:59:06,389 44k INFO ====> Epoch: 6769, cost 14.72 s 2023-09-05 06:59:20,894 44k INFO ====> Epoch: 6770, cost 14.50 s 2023-09-05 06:59:35,525 44k INFO ====> Epoch: 6771, cost 14.63 s 2023-09-05 06:59:41,983 44k INFO Train Epoch: 6772 [38%] 2023-09-05 06:59:41,984 44k INFO Losses: [2.043057680130005, 3.0209858417510986, 9.32340145111084, 15.607707023620605, 0.4162565767765045], step: 142200, lr: 4.289444297415549e-05, reference_loss: 30.411407470703125 2023-09-05 06:59:50,696 44k INFO ====> Epoch: 6772, cost 15.17 s 2023-09-05 07:00:05,494 44k INFO ====> Epoch: 6773, cost 14.80 s 2023-09-05 07:00:20,050 44k INFO ====> Epoch: 6774, cost 14.56 s 2023-09-05 07:00:34,806 44k INFO ====> Epoch: 6775, cost 14.76 s 2023-09-05 07:00:49,399 44k INFO ====> Epoch: 6776, cost 14.59 s 2023-09-05 07:01:03,969 44k INFO ====> Epoch: 6777, cost 14.57 s 2023-09-05 07:01:18,475 44k INFO ====> Epoch: 6778, cost 14.51 s 2023-09-05 07:01:33,116 44k INFO ====> Epoch: 6779, cost 14.64 s 2023-09-05 07:01:47,938 44k INFO ====> Epoch: 6780, cost 14.82 s 2023-09-05 07:02:02,364 44k INFO Train Epoch: 6781 [90%] 2023-09-05 07:02:02,364 44k INFO Losses: [2.2083420753479004, 2.51226806640625, 9.72372817993164, 16.352319717407227, 0.6967223286628723], step: 142400, lr: 4.284621084689768e-05, reference_loss: 31.493379592895508 2023-09-05 07:02:07,927 44k INFO Saving model and optimizer state at iteration 6781 to ./logs\44k\G_142400.pth 2023-09-05 07:02:08,512 44k INFO Saving model and optimizer state at iteration 6781 to ./logs\44k\D_142400.pth 2023-09-05 07:02:09,789 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_140000.pth 2023-09-05 07:02:09,826 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_140000.pth 2023-09-05 07:02:10,290 44k INFO ====> Epoch: 6781, cost 22.35 s 2023-09-05 07:02:24,710 44k INFO ====> Epoch: 6782, cost 14.42 s 2023-09-05 07:02:39,275 44k INFO ====> Epoch: 6783, cost 14.56 s 2023-09-05 07:02:53,822 44k INFO ====> Epoch: 6784, cost 14.55 s 2023-09-05 07:03:08,307 44k INFO ====> Epoch: 6785, cost 14.49 s 2023-09-05 07:03:22,954 44k INFO ====> Epoch: 6786, cost 14.65 s 2023-09-05 07:03:37,642 44k INFO ====> Epoch: 6787, cost 14.69 s 2023-09-05 07:03:52,200 44k INFO ====> Epoch: 6788, cost 14.56 s 2023-09-05 07:04:06,979 44k INFO ====> Epoch: 6789, cost 14.78 s 2023-09-05 07:04:21,555 44k INFO ====> Epoch: 6790, cost 14.58 s 2023-09-05 07:04:28,672 44k INFO Train Epoch: 6791 [43%] 2023-09-05 07:04:28,672 44k INFO Losses: [2.0982468128204346, 2.8553340435028076, 8.212409973144531, 15.486247062683105, 0.5288354158401489], step: 142600, lr: 4.2792683199541163e-05, reference_loss: 29.181072235107422 2023-09-05 07:04:36,473 44k INFO ====> Epoch: 6791, cost 14.92 s 2023-09-05 07:04:51,105 44k INFO ====> Epoch: 6792, cost 14.63 s 2023-09-05 07:05:05,777 44k INFO ====> Epoch: 6793, cost 14.67 s 2023-09-05 07:05:20,467 44k INFO ====> Epoch: 6794, cost 14.69 s 2023-09-05 07:05:34,953 44k INFO ====> Epoch: 6795, cost 14.49 s 2023-09-05 07:05:49,628 44k INFO ====> Epoch: 6796, cost 14.67 s 2023-09-05 07:06:04,333 44k INFO ====> Epoch: 6797, cost 14.71 s 2023-09-05 07:06:19,079 44k INFO ====> Epoch: 6798, cost 14.75 s 2023-09-05 07:06:33,714 44k INFO ====> Epoch: 6799, cost 14.64 s 2023-09-05 07:06:48,270 44k INFO Train Epoch: 6800 [95%] 2023-09-05 07:06:48,271 44k INFO Losses: [1.9835031032562256, 2.8375635147094727, 9.79532241821289, 15.450328826904297, 0.22728751599788666], step: 142800, lr: 4.27445654948066e-05, reference_loss: 30.29400634765625 2023-09-05 07:06:48,538 44k INFO ====> Epoch: 6800, cost 14.82 s 2023-09-05 07:07:03,067 44k INFO ====> Epoch: 6801, cost 14.53 s 2023-09-05 07:07:17,531 44k INFO ====> Epoch: 6802, cost 14.46 s 2023-09-05 07:07:32,017 44k INFO ====> Epoch: 6803, cost 14.49 s 2023-09-05 07:07:47,022 44k INFO ====> Epoch: 6804, cost 15.00 s 2023-09-05 07:08:01,723 44k INFO ====> Epoch: 6805, cost 14.70 s 2023-09-05 07:08:16,385 44k INFO ====> Epoch: 6806, cost 14.66 s 2023-09-05 07:08:30,937 44k INFO ====> Epoch: 6807, cost 14.55 s 2023-09-05 07:08:45,249 44k INFO ====> Epoch: 6808, cost 14.31 s 2023-09-05 07:08:59,755 44k INFO ====> Epoch: 6809, cost 14.51 s 2023-09-05 07:09:07,430 44k INFO Train Epoch: 6810 [48%] 2023-09-05 07:09:07,431 44k INFO Losses: [1.9494285583496094, 3.3696651458740234, 11.057243347167969, 15.833280563354492, 0.5248211622238159], step: 143000, lr: 4.269116483269462e-05, reference_loss: 32.734439849853516 2023-09-05 07:09:14,438 44k INFO ====> Epoch: 6810, cost 14.68 s 2023-09-05 07:09:28,721 44k INFO ====> Epoch: 6811, cost 14.28 s 2023-09-05 07:09:43,197 44k INFO ====> Epoch: 6812, cost 14.48 s 2023-09-05 07:09:57,986 44k INFO ====> Epoch: 6813, cost 14.79 s 2023-09-05 07:10:12,701 44k INFO ====> Epoch: 6814, cost 14.71 s 2023-09-05 07:10:27,242 44k INFO ====> Epoch: 6815, cost 14.54 s 2023-09-05 07:10:41,930 44k INFO ====> Epoch: 6816, cost 14.69 s 2023-09-05 07:10:56,811 44k INFO ====> Epoch: 6817, cost 14.88 s 2023-09-05 07:11:11,482 44k INFO ====> Epoch: 6818, cost 14.67 s 2023-09-05 07:11:25,972 44k INFO ====> Epoch: 6819, cost 14.49 s 2023-09-05 07:11:26,698 44k INFO Train Epoch: 6820 [0%] 2023-09-05 07:11:26,698 44k INFO Losses: [2.3441972732543945, 2.750720739364624, 9.660019874572754, 14.305474281311035, 0.6044870018959045], step: 143200, lr: 4.263783088387545e-05, reference_loss: 29.664899826049805 2023-09-05 07:11:32,143 44k INFO Saving model and optimizer state at iteration 6820 to ./logs\44k\G_143200.pth 2023-09-05 07:11:32,748 44k INFO Saving model and optimizer state at iteration 6820 to ./logs\44k\D_143200.pth 2023-09-05 07:11:33,881 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_140800.pth 2023-09-05 07:11:33,918 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_140800.pth 2023-09-05 07:11:47,828 44k INFO ====> Epoch: 6820, cost 21.86 s 2023-09-05 07:12:02,508 44k INFO ====> Epoch: 6821, cost 14.68 s 2023-09-05 07:12:17,231 44k INFO ====> Epoch: 6822, cost 14.72 s 2023-09-05 07:12:31,672 44k INFO ====> Epoch: 6823, cost 14.44 s 2023-09-05 07:12:46,366 44k INFO ====> Epoch: 6824, cost 14.69 s 2023-09-05 07:13:00,734 44k INFO ====> Epoch: 6825, cost 14.37 s 2023-09-05 07:13:15,410 44k INFO ====> Epoch: 6826, cost 14.68 s 2023-09-05 07:13:29,838 44k INFO ====> Epoch: 6827, cost 14.43 s 2023-09-05 07:13:44,319 44k INFO ====> Epoch: 6828, cost 14.48 s 2023-09-05 07:13:52,924 44k INFO Train Epoch: 6829 [52%] 2023-09-05 07:13:52,925 44k INFO Losses: [2.428701877593994, 2.4126052856445312, 7.575302600860596, 14.912001609802246, 0.6131961941719055], step: 143400, lr: 4.258988730091699e-05, reference_loss: 27.94180679321289 2023-09-05 07:13:59,350 44k INFO ====> Epoch: 6829, cost 15.03 s 2023-09-05 07:14:13,957 44k INFO ====> Epoch: 6830, cost 14.61 s 2023-09-05 07:14:28,220 44k INFO ====> Epoch: 6831, cost 14.26 s 2023-09-05 07:14:42,616 44k INFO ====> Epoch: 6832, cost 14.40 s 2023-09-05 07:14:57,102 44k INFO ====> Epoch: 6833, cost 14.49 s 2023-09-05 07:15:11,486 44k INFO ====> Epoch: 6834, cost 14.38 s 2023-09-05 07:15:25,846 44k INFO ====> Epoch: 6835, cost 14.36 s 2023-09-05 07:15:40,282 44k INFO ====> Epoch: 6836, cost 14.44 s 2023-09-05 07:15:54,768 44k INFO ====> Epoch: 6837, cost 14.49 s 2023-09-05 07:16:09,507 44k INFO ====> Epoch: 6838, cost 14.74 s 2023-09-05 07:16:10,955 44k INFO Train Epoch: 6839 [5%] 2023-09-05 07:16:10,955 44k INFO Losses: [2.2564101219177246, 2.9457993507385254, 9.865486145019531, 14.4323148727417, 0.5111978650093079], step: 143600, lr: 4.253667987782551e-05, reference_loss: 30.01120948791504 2023-09-05 07:16:24,417 44k INFO ====> Epoch: 6839, cost 14.91 s 2023-09-05 07:16:38,650 44k INFO ====> Epoch: 6840, cost 14.23 s 2023-09-05 07:16:53,221 44k INFO ====> Epoch: 6841, cost 14.57 s 2023-09-05 07:17:07,926 44k INFO ====> Epoch: 6842, cost 14.71 s 2023-09-05 07:17:22,392 44k INFO ====> Epoch: 6843, cost 14.47 s 2023-09-05 07:17:36,651 44k INFO ====> Epoch: 6844, cost 14.26 s 2023-09-05 07:17:51,454 44k INFO ====> Epoch: 6845, cost 14.80 s 2023-09-05 07:18:06,123 44k INFO ====> Epoch: 6846, cost 14.67 s 2023-09-05 07:18:20,715 44k INFO ====> Epoch: 6847, cost 14.59 s 2023-09-05 07:18:29,839 44k INFO Train Epoch: 6848 [57%] 2023-09-05 07:18:29,839 44k INFO Losses: [2.132793664932251, 3.1237127780914307, 11.755851745605469, 16.469526290893555, 0.47404757142066956], step: 143800, lr: 4.2488850032868005e-05, reference_loss: 33.9559326171875 2023-09-05 07:18:35,377 44k INFO ====> Epoch: 6848, cost 14.66 s 2023-09-05 07:18:49,900 44k INFO ====> Epoch: 6849, cost 14.52 s 2023-09-05 07:19:04,510 44k INFO ====> Epoch: 6850, cost 14.61 s 2023-09-05 07:19:19,123 44k INFO ====> Epoch: 6851, cost 14.61 s 2023-09-05 07:19:33,443 44k INFO ====> Epoch: 6852, cost 14.32 s 2023-09-05 07:19:48,047 44k INFO ====> Epoch: 6853, cost 14.60 s 2023-09-05 07:20:02,918 44k INFO ====> Epoch: 6854, cost 14.87 s 2023-09-05 07:20:17,434 44k INFO ====> Epoch: 6855, cost 14.52 s 2023-09-05 07:20:31,893 44k INFO ====> Epoch: 6856, cost 14.46 s 2023-09-05 07:20:46,530 44k INFO ====> Epoch: 6857, cost 14.64 s 2023-09-05 07:20:48,655 44k INFO Train Epoch: 6858 [10%] 2023-09-05 07:20:48,655 44k INFO Losses: [2.2167983055114746, 2.7486460208892822, 10.070927619934082, 15.165024757385254, 0.6934496760368347], step: 144000, lr: 4.243576883534342e-05, reference_loss: 30.894845962524414 2023-09-05 07:20:54,244 44k INFO Saving model and optimizer state at iteration 6858 to ./logs\44k\G_144000.pth 2023-09-05 07:20:54,860 44k INFO Saving model and optimizer state at iteration 6858 to ./logs\44k\D_144000.pth 2023-09-05 07:20:55,980 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_141600.pth 2023-09-05 07:20:56,016 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_141600.pth 2023-09-05 07:21:08,483 44k INFO ====> Epoch: 6858, cost 21.95 s 2023-09-05 07:21:22,970 44k INFO ====> Epoch: 6859, cost 14.49 s 2023-09-05 07:21:37,349 44k INFO ====> Epoch: 6860, cost 14.38 s 2023-09-05 07:21:51,846 44k INFO ====> Epoch: 6861, cost 14.50 s 2023-09-05 07:22:06,444 44k INFO ====> Epoch: 6862, cost 14.60 s 2023-09-05 07:22:20,931 44k INFO ====> Epoch: 6863, cost 14.49 s 2023-09-05 07:22:35,445 44k INFO ====> Epoch: 6864, cost 14.51 s 2023-09-05 07:22:50,006 44k INFO ====> Epoch: 6865, cost 14.56 s 2023-09-05 07:23:04,696 44k INFO ====> Epoch: 6866, cost 14.69 s 2023-09-05 07:23:14,622 44k INFO Train Epoch: 6867 [62%] 2023-09-05 07:23:14,622 44k INFO Losses: [2.252082109451294, 2.51385235786438, 10.00996208190918, 15.673561096191406, 0.4313600957393646], step: 144200, lr: 4.23880524585628e-05, reference_loss: 30.880817413330078 2023-09-05 07:23:19,499 44k INFO ====> Epoch: 6867, cost 14.80 s 2023-09-05 07:23:33,792 44k INFO ====> Epoch: 6868, cost 14.29 s 2023-09-05 07:23:48,491 44k INFO ====> Epoch: 6869, cost 14.70 s 2023-09-05 07:24:03,243 44k INFO ====> Epoch: 6870, cost 14.75 s 2023-09-05 07:24:17,936 44k INFO ====> Epoch: 6871, cost 14.69 s 2023-09-05 07:24:32,445 44k INFO ====> Epoch: 6872, cost 14.51 s 2023-09-05 07:24:47,012 44k INFO ====> Epoch: 6873, cost 14.57 s 2023-09-05 07:25:01,569 44k INFO ====> Epoch: 6874, cost 14.56 s 2023-09-05 07:25:16,123 44k INFO ====> Epoch: 6875, cost 14.55 s 2023-09-05 07:25:30,447 44k INFO ====> Epoch: 6876, cost 14.32 s 2023-09-05 07:25:33,256 44k INFO Train Epoch: 6877 [14%] 2023-09-05 07:25:33,256 44k INFO Losses: [2.210381507873535, 2.7077460289001465, 9.530384063720703, 16.14811134338379, 0.5251794457435608], step: 144400, lr: 4.2335097187156435e-05, reference_loss: 31.121801376342773 2023-09-05 07:25:45,183 44k INFO ====> Epoch: 6877, cost 14.74 s 2023-09-05 07:25:59,885 44k INFO ====> Epoch: 6878, cost 14.70 s 2023-09-05 07:26:14,528 44k INFO ====> Epoch: 6879, cost 14.64 s 2023-09-05 07:26:28,856 44k INFO ====> Epoch: 6880, cost 14.33 s 2023-09-05 07:26:43,446 44k INFO ====> Epoch: 6881, cost 14.59 s 2023-09-05 07:26:58,098 44k INFO ====> Epoch: 6882, cost 14.65 s 2023-09-05 07:27:12,677 44k INFO ====> Epoch: 6883, cost 14.58 s 2023-09-05 07:27:27,075 44k INFO ====> Epoch: 6884, cost 14.40 s 2023-09-05 07:27:41,584 44k INFO ====> Epoch: 6885, cost 14.51 s 2023-09-05 07:27:52,164 44k INFO Train Epoch: 6886 [67%] 2023-09-05 07:27:52,166 44k INFO Losses: [2.5661981105804443, 2.171708583831787, 6.89570426940918, 12.512723922729492, 0.48506245017051697], step: 144600, lr: 4.228749400936875e-05, reference_loss: 24.631397247314453 2023-09-05 07:27:56,389 44k INFO ====> Epoch: 6886, cost 14.80 s 2023-09-05 07:28:10,951 44k INFO ====> Epoch: 6887, cost 14.56 s 2023-09-05 07:28:25,484 44k INFO ====> Epoch: 6888, cost 14.53 s 2023-09-05 07:28:40,006 44k INFO ====> Epoch: 6889, cost 14.52 s 2023-09-05 07:28:54,848 44k INFO ====> Epoch: 6890, cost 14.84 s 2023-09-05 07:29:09,483 44k INFO ====> Epoch: 6891, cost 14.64 s 2023-09-05 07:29:23,867 44k INFO ====> Epoch: 6892, cost 14.38 s 2023-09-05 07:29:38,199 44k INFO ====> Epoch: 6893, cost 14.33 s 2023-09-05 07:29:52,868 44k INFO ====> Epoch: 6894, cost 14.67 s 2023-09-05 07:30:07,625 44k INFO ====> Epoch: 6895, cost 14.76 s 2023-09-05 07:30:11,299 44k INFO Train Epoch: 6896 [19%] 2023-09-05 07:30:11,300 44k INFO Losses: [2.5025599002838135, 2.2362120151519775, 7.607327938079834, 14.21854305267334, 0.6529604196548462], step: 144800, lr: 4.223466436534228e-05, reference_loss: 27.21760368347168 2023-09-05 07:30:16,711 44k INFO Saving model and optimizer state at iteration 6896 to ./logs\44k\G_144800.pth 2023-09-05 07:30:17,327 44k INFO Saving model and optimizer state at iteration 6896 to ./logs\44k\D_144800.pth 2023-09-05 07:30:18,559 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_142400.pth 2023-09-05 07:30:18,597 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_142400.pth 2023-09-05 07:30:29,840 44k INFO ====> Epoch: 6896, cost 22.21 s 2023-09-05 07:30:44,420 44k INFO ====> Epoch: 6897, cost 14.58 s 2023-09-05 07:30:59,004 44k INFO ====> Epoch: 6898, cost 14.58 s 2023-09-05 07:31:13,765 44k INFO ====> Epoch: 6899, cost 14.76 s 2023-09-05 07:31:28,151 44k INFO ====> Epoch: 6900, cost 14.39 s 2023-09-05 07:31:42,624 44k INFO ====> Epoch: 6901, cost 14.47 s 2023-09-05 07:31:57,272 44k INFO ====> Epoch: 6902, cost 14.65 s 2023-09-05 07:32:11,920 44k INFO ====> Epoch: 6903, cost 14.65 s 2023-09-05 07:32:26,507 44k INFO ====> Epoch: 6904, cost 14.59 s 2023-09-05 07:32:37,836 44k INFO Train Epoch: 6905 [71%] 2023-09-05 07:32:37,836 44k INFO Losses: [2.1880812644958496, 2.6107778549194336, 8.55923843383789, 14.00640869140625, 0.6950183510780334], step: 145000, lr: 4.218717411800213e-05, reference_loss: 28.059524536132812 2023-09-05 07:32:41,350 44k INFO ====> Epoch: 6905, cost 14.84 s 2023-09-05 07:32:55,811 44k INFO ====> Epoch: 6906, cost 14.46 s 2023-09-05 07:33:10,372 44k INFO ====> Epoch: 6907, cost 14.56 s 2023-09-05 07:33:24,719 44k INFO ====> Epoch: 6908, cost 14.35 s 2023-09-05 07:33:39,182 44k INFO ====> Epoch: 6909, cost 14.46 s 2023-09-05 07:33:53,819 44k INFO ====> Epoch: 6910, cost 14.64 s 2023-09-05 07:34:08,539 44k INFO ====> Epoch: 6911, cost 14.72 s 2023-09-05 07:34:23,152 44k INFO ====> Epoch: 6912, cost 14.61 s 2023-09-05 07:34:37,472 44k INFO ====> Epoch: 6913, cost 14.32 s 2023-09-05 07:34:52,212 44k INFO ====> Epoch: 6914, cost 14.74 s 2023-09-05 07:34:56,515 44k INFO Train Epoch: 6915 [24%] 2023-09-05 07:34:56,515 44k INFO Losses: [2.29329514503479, 2.625171422958374, 9.806676864624023, 15.827430725097656, 0.4658510684967041], step: 145200, lr: 4.213446980332595e-05, reference_loss: 31.01842498779297 2023-09-05 07:35:07,246 44k INFO ====> Epoch: 6915, cost 15.03 s 2023-09-05 07:35:21,857 44k INFO ====> Epoch: 6916, cost 14.61 s 2023-09-05 07:35:36,136 44k INFO ====> Epoch: 6917, cost 14.28 s 2023-09-05 07:35:50,831 44k INFO ====> Epoch: 6918, cost 14.69 s 2023-09-05 07:36:05,588 44k INFO ====> Epoch: 6919, cost 14.76 s 2023-09-05 07:36:20,140 44k INFO ====> Epoch: 6920, cost 14.55 s 2023-09-05 07:36:34,572 44k INFO ====> Epoch: 6921, cost 14.43 s 2023-09-05 07:36:49,033 44k INFO ====> Epoch: 6922, cost 14.46 s 2023-09-05 07:37:03,670 44k INFO ====> Epoch: 6923, cost 14.64 s 2023-09-05 07:37:15,566 44k INFO Train Epoch: 6924 [76%] 2023-09-05 07:37:15,566 44k INFO Losses: [2.0599465370178223, 2.8152661323547363, 8.363089561462402, 12.76898193359375, 0.4574151039123535], step: 145400, lr: 4.208709221852505e-05, reference_loss: 26.464698791503906 2023-09-05 07:37:18,320 44k INFO ====> Epoch: 6924, cost 14.65 s 2023-09-05 07:37:32,695 44k INFO ====> Epoch: 6925, cost 14.37 s 2023-09-05 07:37:47,145 44k INFO ====> Epoch: 6926, cost 14.45 s 2023-09-05 07:38:01,983 44k INFO ====> Epoch: 6927, cost 14.84 s 2023-09-05 07:38:16,600 44k INFO ====> Epoch: 6928, cost 14.62 s 2023-09-05 07:38:31,000 44k INFO ====> Epoch: 6929, cost 14.40 s 2023-09-05 07:38:45,562 44k INFO ====> Epoch: 6930, cost 14.56 s 2023-09-05 07:39:00,350 44k INFO ====> Epoch: 6931, cost 14.79 s 2023-09-05 07:39:14,894 44k INFO ====> Epoch: 6932, cost 14.54 s 2023-09-05 07:39:29,247 44k INFO ====> Epoch: 6933, cost 14.35 s 2023-09-05 07:39:34,145 44k INFO Train Epoch: 6934 [29%] 2023-09-05 07:39:34,146 44k INFO Losses: [2.13668155670166, 2.7003262042999268, 10.265376091003418, 16.899423599243164, 0.681147038936615], step: 145600, lr: 4.2034512935876585e-05, reference_loss: 32.68295669555664 2023-09-05 07:39:39,541 44k INFO Saving model and optimizer state at iteration 6934 to ./logs\44k\G_145600.pth 2023-09-05 07:39:40,207 44k INFO Saving model and optimizer state at iteration 6934 to ./logs\44k\D_145600.pth 2023-09-05 07:39:41,340 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_143200.pth 2023-09-05 07:39:41,377 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_143200.pth 2023-09-05 07:39:50,984 44k INFO ====> Epoch: 6934, cost 21.74 s 2023-09-05 07:40:05,766 44k INFO ====> Epoch: 6935, cost 14.78 s 2023-09-05 07:40:20,253 44k INFO ====> Epoch: 6936, cost 14.49 s 2023-09-05 07:40:34,859 44k INFO ====> Epoch: 6937, cost 14.61 s 2023-09-05 07:40:49,469 44k INFO ====> Epoch: 6938, cost 14.61 s 2023-09-05 07:41:04,279 44k INFO ====> Epoch: 6939, cost 14.81 s 2023-09-05 07:41:18,684 44k INFO ====> Epoch: 6940, cost 14.41 s 2023-09-05 07:41:33,054 44k INFO ====> Epoch: 6941, cost 14.37 s 2023-09-05 07:41:47,474 44k INFO ====> Epoch: 6942, cost 14.42 s 2023-09-05 07:42:00,392 44k INFO Train Epoch: 6943 [81%] 2023-09-05 07:42:00,393 44k INFO Losses: [2.1158673763275146, 2.6914384365081787, 9.321364402770996, 15.698126792907715, 0.6049658060073853], step: 145800, lr: 4.1987247746342236e-05, reference_loss: 30.4317626953125 2023-09-05 07:42:02,506 44k INFO ====> Epoch: 6943, cost 15.03 s 2023-09-05 07:42:17,102 44k INFO ====> Epoch: 6944, cost 14.60 s 2023-09-05 07:42:31,382 44k INFO ====> Epoch: 6945, cost 14.28 s 2023-09-05 07:42:45,786 44k INFO ====> Epoch: 6946, cost 14.40 s 2023-09-05 07:43:00,355 44k INFO ====> Epoch: 6947, cost 14.57 s 2023-09-05 07:43:14,762 44k INFO ====> Epoch: 6948, cost 14.41 s 2023-09-05 07:43:29,142 44k INFO ====> Epoch: 6949, cost 14.38 s 2023-09-05 07:43:43,639 44k INFO ====> Epoch: 6950, cost 14.50 s 2023-09-05 07:43:58,416 44k INFO ====> Epoch: 6951, cost 14.78 s 2023-09-05 07:44:13,035 44k INFO ====> Epoch: 6952, cost 14.62 s 2023-09-05 07:44:18,710 44k INFO Train Epoch: 6953 [33%] 2023-09-05 07:44:18,711 44k INFO Losses: [2.3127994537353516, 2.6057283878326416, 9.932605743408203, 15.67048454284668, 0.6458718180656433], step: 146000, lr: 4.1934793199104254e-05, reference_loss: 31.167490005493164 2023-09-05 07:44:27,831 44k INFO ====> Epoch: 6953, cost 14.80 s 2023-09-05 07:44:42,390 44k INFO ====> Epoch: 6954, cost 14.56 s 2023-09-05 07:44:56,993 44k INFO ====> Epoch: 6955, cost 14.60 s 2023-09-05 07:45:11,500 44k INFO ====> Epoch: 6956, cost 14.51 s 2023-09-05 07:45:25,979 44k INFO ====> Epoch: 6957, cost 14.48 s 2023-09-05 07:45:40,455 44k INFO ====> Epoch: 6958, cost 14.48 s 2023-09-05 07:45:55,103 44k INFO ====> Epoch: 6959, cost 14.65 s 2023-09-05 07:46:09,831 44k INFO ====> Epoch: 6960, cost 14.73 s 2023-09-05 07:46:24,287 44k INFO ====> Epoch: 6961, cost 14.46 s 2023-09-05 07:46:37,681 44k INFO Train Epoch: 6962 [86%] 2023-09-05 07:46:37,681 44k INFO Losses: [2.235481023788452, 2.6695613861083984, 9.559260368347168, 14.417813301086426, 0.6349221467971802], step: 146200, lr: 4.188764013819779e-05, reference_loss: 29.517038345336914 2023-09-05 07:46:39,132 44k INFO ====> Epoch: 6962, cost 14.84 s 2023-09-05 07:46:53,891 44k INFO ====> Epoch: 6963, cost 14.76 s 2023-09-05 07:47:08,439 44k INFO ====> Epoch: 6964, cost 14.55 s 2023-09-05 07:47:22,872 44k INFO ====> Epoch: 6965, cost 14.43 s 2023-09-05 07:47:37,334 44k INFO ====> Epoch: 6966, cost 14.46 s 2023-09-05 07:47:52,029 44k INFO ====> Epoch: 6967, cost 14.70 s 2023-09-05 07:48:06,718 44k INFO ====> Epoch: 6968, cost 14.69 s 2023-09-05 07:48:21,251 44k INFO ====> Epoch: 6969, cost 14.53 s 2023-09-05 07:48:35,605 44k INFO ====> Epoch: 6970, cost 14.35 s 2023-09-05 07:48:50,236 44k INFO ====> Epoch: 6971, cost 14.63 s 2023-09-05 07:48:56,650 44k INFO Train Epoch: 6972 [38%] 2023-09-05 07:48:56,651 44k INFO Losses: [2.1662118434906006, 2.5014030933380127, 7.902451038360596, 13.42451000213623, 0.5996506214141846], step: 146400, lr: 4.183531003045674e-05, reference_loss: 26.594226837158203 2023-09-05 07:49:02,105 44k INFO Saving model and optimizer state at iteration 6972 to ./logs\44k\G_146400.pth 2023-09-05 07:49:02,783 44k INFO Saving model and optimizer state at iteration 6972 to ./logs\44k\D_146400.pth 2023-09-05 07:49:03,924 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_144000.pth 2023-09-05 07:49:03,961 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_144000.pth 2023-09-05 07:49:12,141 44k INFO ====> Epoch: 6972, cost 21.91 s 2023-09-05 07:49:26,502 44k INFO ====> Epoch: 6973, cost 14.36 s 2023-09-05 07:49:40,886 44k INFO ====> Epoch: 6974, cost 14.38 s 2023-09-05 07:49:55,568 44k INFO ====> Epoch: 6975, cost 14.68 s 2023-09-05 07:50:10,360 44k INFO ====> Epoch: 6976, cost 14.79 s 2023-09-05 07:50:24,959 44k INFO ====> Epoch: 6977, cost 14.60 s 2023-09-05 07:50:39,257 44k INFO ====> Epoch: 6978, cost 14.30 s 2023-09-05 07:50:53,820 44k INFO ====> Epoch: 6979, cost 14.56 s 2023-09-05 07:51:08,308 44k INFO ====> Epoch: 6980, cost 14.49 s 2023-09-05 07:51:22,293 44k INFO Train Epoch: 6981 [90%] 2023-09-05 07:51:22,293 44k INFO Losses: [2.0825376510620117, 2.693924903869629, 9.978569030761719, 15.84592056274414, 0.7695058584213257], step: 146600, lr: 4.178826883217204e-05, reference_loss: 31.370458602905273 2023-09-05 07:51:22,970 44k INFO ====> Epoch: 6981, cost 14.66 s 2023-09-05 07:51:37,446 44k INFO ====> Epoch: 6982, cost 14.48 s 2023-09-05 07:51:51,886 44k INFO ====> Epoch: 6983, cost 14.44 s 2023-09-05 07:52:06,340 44k INFO ====> Epoch: 6984, cost 14.45 s 2023-09-05 07:52:20,877 44k INFO ====> Epoch: 6985, cost 14.54 s 2023-09-05 07:52:35,328 44k INFO ====> Epoch: 6986, cost 14.45 s 2023-09-05 07:52:49,875 44k INFO ====> Epoch: 6987, cost 14.55 s 2023-09-05 07:53:04,524 44k INFO ====> Epoch: 6988, cost 14.65 s 2023-09-05 07:53:19,056 44k INFO ====> Epoch: 6989, cost 14.53 s 2023-09-05 07:53:33,325 44k INFO ====> Epoch: 6990, cost 14.27 s 2023-09-05 07:53:40,287 44k INFO Train Epoch: 6991 [43%] 2023-09-05 07:53:40,287 44k INFO Losses: [2.2464537620544434, 2.6120967864990234, 9.67262077331543, 15.23388671875, 0.5119887590408325], step: 146800, lr: 4.173606286871635e-05, reference_loss: 30.277048110961914 2023-09-05 07:53:48,165 44k INFO ====> Epoch: 6991, cost 14.84 s 2023-09-05 07:54:03,121 44k INFO ====> Epoch: 6992, cost 14.96 s 2023-09-05 07:54:17,668 44k INFO ====> Epoch: 6993, cost 14.55 s 2023-09-05 07:54:32,149 44k INFO ====> Epoch: 6994, cost 14.48 s 2023-09-05 07:54:46,704 44k INFO ====> Epoch: 6995, cost 14.55 s 2023-09-05 07:55:01,504 44k INFO ====> Epoch: 6996, cost 14.80 s 2023-09-05 07:55:16,026 44k INFO ====> Epoch: 6997, cost 14.52 s 2023-09-05 07:55:30,290 44k INFO ====> Epoch: 6998, cost 14.26 s 2023-09-05 07:55:44,821 44k INFO ====> Epoch: 6999, cost 14.53 s 2023-09-05 07:55:59,532 44k INFO Train Epoch: 7000 [95%] 2023-09-05 07:55:59,533 44k INFO Losses: [2.0785794258117676, 3.0886306762695312, 10.275372505187988, 14.234593391418457, 0.4683391749858856], step: 147000, lr: 4.1689133267678346e-05, reference_loss: 30.145517349243164 2023-09-05 07:55:59,803 44k INFO ====> Epoch: 7000, cost 14.98 s 2023-09-05 07:56:14,394 44k INFO ====> Epoch: 7001, cost 14.59 s 2023-09-05 07:56:28,989 44k INFO ====> Epoch: 7002, cost 14.60 s 2023-09-05 07:56:43,579 44k INFO ====> Epoch: 7003, cost 14.59 s 2023-09-05 07:56:58,184 44k INFO ====> Epoch: 7004, cost 14.61 s 2023-09-05 07:57:12,708 44k INFO ====> Epoch: 7005, cost 14.52 s 2023-09-05 07:57:27,561 44k INFO ====> Epoch: 7006, cost 14.85 s 2023-09-05 07:57:42,000 44k INFO ====> Epoch: 7007, cost 14.44 s 2023-09-05 07:57:56,655 44k INFO ====> Epoch: 7008, cost 14.65 s 2023-09-05 07:58:11,369 44k INFO ====> Epoch: 7009, cost 14.71 s 2023-09-05 07:58:18,995 44k INFO Train Epoch: 7010 [48%] 2023-09-05 07:58:18,995 44k INFO Losses: [2.1355395317077637, 2.7051491737365723, 11.376480102539062, 16.94822883605957, 0.6856611371040344], step: 147200, lr: 4.163705115399681e-05, reference_loss: 33.85105895996094 2023-09-05 07:58:24,435 44k INFO Saving model and optimizer state at iteration 7010 to ./logs\44k\G_147200.pth 2023-09-05 07:58:25,133 44k INFO Saving model and optimizer state at iteration 7010 to ./logs\44k\D_147200.pth 2023-09-05 07:58:26,211 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_144800.pth 2023-09-05 07:58:26,252 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_144800.pth 2023-09-05 07:58:32,943 44k INFO ====> Epoch: 7010, cost 21.57 s 2023-09-05 07:58:47,485 44k INFO ====> Epoch: 7011, cost 14.54 s 2023-09-05 07:59:02,125 44k INFO ====> Epoch: 7012, cost 14.64 s 2023-09-05 07:59:16,679 44k INFO ====> Epoch: 7013, cost 14.55 s 2023-09-05 07:59:31,116 44k INFO ====> Epoch: 7014, cost 14.44 s 2023-09-05 07:59:45,672 44k INFO ====> Epoch: 7015, cost 14.56 s 2023-09-05 08:00:00,366 44k INFO ====> Epoch: 7016, cost 14.69 s 2023-09-05 08:00:15,082 44k INFO ====> Epoch: 7017, cost 14.72 s 2023-09-05 08:00:29,462 44k INFO ====> Epoch: 7018, cost 14.38 s 2023-09-05 08:00:44,079 44k INFO ====> Epoch: 7019, cost 14.62 s 2023-09-05 08:00:44,749 44k INFO Train Epoch: 7020 [0%] 2023-09-05 08:00:44,750 44k INFO Losses: [2.164551019668579, 3.0863122940063477, 8.587013244628906, 13.535100936889648, 0.5555362105369568], step: 147400, lr: 4.158503410634934e-05, reference_loss: 27.92851448059082 2023-09-05 08:00:58,896 44k INFO ====> Epoch: 7020, cost 14.82 s 2023-09-05 08:01:13,419 44k INFO ====> Epoch: 7021, cost 14.52 s 2023-09-05 08:01:27,671 44k INFO ====> Epoch: 7022, cost 14.25 s 2023-09-05 08:01:42,071 44k INFO ====> Epoch: 7023, cost 14.40 s 2023-09-05 08:01:56,736 44k INFO ====> Epoch: 7024, cost 14.67 s 2023-09-05 08:02:11,451 44k INFO ====> Epoch: 7025, cost 14.71 s 2023-09-05 08:02:25,980 44k INFO ====> Epoch: 7026, cost 14.53 s 2023-09-05 08:02:40,457 44k INFO ====> Epoch: 7027, cost 14.48 s 2023-09-05 08:02:54,965 44k INFO ====> Epoch: 7028, cost 14.51 s 2023-09-05 08:03:03,485 44k INFO Train Epoch: 7029 [52%] 2023-09-05 08:03:03,485 44k INFO Losses: [2.1737380027770996, 2.896919012069702, 8.948941230773926, 16.193042755126953, 0.6734609007835388], step: 147600, lr: 4.1538274327740104e-05, reference_loss: 30.88610076904297 2023-09-05 08:03:09,903 44k INFO ====> Epoch: 7029, cost 14.94 s 2023-09-05 08:03:24,330 44k INFO ====> Epoch: 7030, cost 14.43 s 2023-09-05 08:03:38,868 44k INFO ====> Epoch: 7031, cost 14.54 s 2023-09-05 08:03:53,605 44k INFO ====> Epoch: 7032, cost 14.74 s 2023-09-05 08:04:08,297 44k INFO ====> Epoch: 7033, cost 14.69 s 2023-09-05 08:04:22,874 44k INFO ====> Epoch: 7034, cost 14.58 s 2023-09-05 08:04:37,266 44k INFO ====> Epoch: 7035, cost 14.39 s 2023-09-05 08:04:51,756 44k INFO ====> Epoch: 7036, cost 14.49 s 2023-09-05 08:05:06,536 44k INFO ====> Epoch: 7037, cost 14.78 s 2023-09-05 08:05:21,045 44k INFO ====> Epoch: 7038, cost 14.51 s 2023-09-05 08:05:22,458 44k INFO Train Epoch: 7039 [5%] 2023-09-05 08:05:22,458 44k INFO Losses: [2.1546356678009033, 2.7538111209869385, 8.882879257202148, 16.529924392700195, 0.559478223323822], step: 147800, lr: 4.148638068169615e-05, reference_loss: 30.88072967529297 2023-09-05 08:05:35,705 44k INFO ====> Epoch: 7039, cost 14.66 s 2023-09-05 08:05:50,234 44k INFO ====> Epoch: 7040, cost 14.53 s 2023-09-05 08:06:04,858 44k INFO ====> Epoch: 7041, cost 14.62 s 2023-09-05 08:06:19,393 44k INFO ====> Epoch: 7042, cost 14.53 s 2023-09-05 08:06:33,901 44k INFO ====> Epoch: 7043, cost 14.51 s 2023-09-05 08:06:48,370 44k INFO ====> Epoch: 7044, cost 14.47 s 2023-09-05 08:07:02,861 44k INFO ====> Epoch: 7045, cost 14.49 s 2023-09-05 08:07:17,422 44k INFO ====> Epoch: 7046, cost 14.56 s 2023-09-05 08:07:31,886 44k INFO ====> Epoch: 7047, cost 14.46 s 2023-09-05 08:07:40,919 44k INFO Train Epoch: 7048 [57%] 2023-09-05 08:07:40,919 44k INFO Losses: [2.2055768966674805, 2.6659226417541504, 9.418936729431152, 14.811479568481445, 0.7653735280036926], step: 148000, lr: 4.143973183271328e-05, reference_loss: 29.86728858947754 2023-09-05 08:07:46,372 44k INFO Saving model and optimizer state at iteration 7048 to ./logs\44k\G_148000.pth 2023-09-05 08:07:47,057 44k INFO Saving model and optimizer state at iteration 7048 to ./logs\44k\D_148000.pth 2023-09-05 08:07:48,168 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_145600.pth 2023-09-05 08:07:48,204 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_145600.pth 2023-09-05 08:07:53,623 44k INFO ====> Epoch: 7048, cost 21.74 s 2023-09-05 08:08:08,362 44k INFO ====> Epoch: 7049, cost 14.74 s 2023-09-05 08:08:22,938 44k INFO ====> Epoch: 7050, cost 14.58 s 2023-09-05 08:08:37,383 44k INFO ====> Epoch: 7051, cost 14.45 s 2023-09-05 08:08:51,890 44k INFO ====> Epoch: 7052, cost 14.51 s 2023-09-05 08:09:06,463 44k INFO ====> Epoch: 7053, cost 14.57 s 2023-09-05 08:09:20,905 44k INFO ====> Epoch: 7054, cost 14.44 s 2023-09-05 08:09:35,387 44k INFO ====> Epoch: 7055, cost 14.48 s 2023-09-05 08:09:49,878 44k INFO ====> Epoch: 7056, cost 14.49 s 2023-09-05 08:10:04,406 44k INFO ====> Epoch: 7057, cost 14.53 s 2023-09-05 08:10:06,518 44k INFO Train Epoch: 7058 [10%] 2023-09-05 08:10:06,518 44k INFO Losses: [2.155583620071411, 2.841160774230957, 8.597319602966309, 14.087331771850586, 0.6393634080886841], step: 148200, lr: 4.138796129552349e-05, reference_loss: 28.32076072692871 2023-09-05 08:10:19,106 44k INFO ====> Epoch: 7058, cost 14.70 s 2023-09-05 08:10:33,514 44k INFO ====> Epoch: 7059, cost 14.41 s 2023-09-05 08:10:48,056 44k INFO ====> Epoch: 7060, cost 14.54 s 2023-09-05 08:11:02,727 44k INFO ====> Epoch: 7061, cost 14.67 s 2023-09-05 08:11:17,355 44k INFO ====> Epoch: 7062, cost 14.63 s 2023-09-05 08:11:31,769 44k INFO ====> Epoch: 7063, cost 14.41 s 2023-09-05 08:11:46,180 44k INFO ====> Epoch: 7064, cost 14.41 s 2023-09-05 08:12:00,860 44k INFO ====> Epoch: 7065, cost 14.68 s 2023-09-05 08:12:15,522 44k INFO ====> Epoch: 7066, cost 14.66 s 2023-09-05 08:12:25,535 44k INFO Train Epoch: 7067 [62%] 2023-09-05 08:12:25,535 44k INFO Losses: [2.2898623943328857, 2.446915626525879, 9.31112289428711, 16.318538665771484, 0.5449123978614807], step: 148400, lr: 4.134142311300531e-05, reference_loss: 30.911352157592773 2023-09-05 08:12:30,495 44k INFO ====> Epoch: 7067, cost 14.97 s 2023-09-05 08:12:44,957 44k INFO ====> Epoch: 7068, cost 14.46 s 2023-09-05 08:12:59,590 44k INFO ====> Epoch: 7069, cost 14.63 s 2023-09-05 08:13:14,057 44k INFO ====> Epoch: 7070, cost 14.47 s 2023-09-05 08:13:28,382 44k INFO ====> Epoch: 7071, cost 14.33 s 2023-09-05 08:13:42,836 44k INFO ====> Epoch: 7072, cost 14.45 s 2023-09-05 08:13:57,537 44k INFO ====> Epoch: 7073, cost 14.70 s 2023-09-05 08:14:12,156 44k INFO ====> Epoch: 7074, cost 14.62 s 2023-09-05 08:14:26,528 44k INFO ====> Epoch: 7075, cost 14.37 s 2023-09-05 08:14:40,967 44k INFO ====> Epoch: 7076, cost 14.44 s 2023-09-05 08:14:43,727 44k INFO Train Epoch: 7077 [14%] 2023-09-05 08:14:43,727 44k INFO Losses: [2.2396240234375, 2.609135150909424, 10.769947052001953, 15.288247108459473, 0.5831061005592346], step: 148600, lr: 4.1289775392614886e-05, reference_loss: 31.49005889892578 2023-09-05 08:14:55,710 44k INFO ====> Epoch: 7077, cost 14.74 s 2023-09-05 08:15:10,383 44k INFO ====> Epoch: 7078, cost 14.67 s 2023-09-05 08:15:24,713 44k INFO ====> Epoch: 7079, cost 14.33 s 2023-09-05 08:15:39,136 44k INFO ====> Epoch: 7080, cost 14.42 s 2023-09-05 08:15:53,626 44k INFO ====> Epoch: 7081, cost 14.49 s 2023-09-05 08:16:08,126 44k INFO ====> Epoch: 7082, cost 14.50 s 2023-09-05 08:16:22,728 44k INFO ====> Epoch: 7083, cost 14.60 s 2023-09-05 08:16:37,133 44k INFO ====> Epoch: 7084, cost 14.41 s 2023-09-05 08:16:51,660 44k INFO ====> Epoch: 7085, cost 14.53 s 2023-09-05 08:17:02,318 44k INFO Train Epoch: 7086 [67%] 2023-09-05 08:17:02,318 44k INFO Losses: [2.350421190261841, 2.44431734085083, 9.793731689453125, 16.69504165649414, 0.5643038749694824], step: 148800, lr: 4.1243347614024e-05, reference_loss: 31.847816467285156 2023-09-05 08:17:07,973 44k INFO Saving model and optimizer state at iteration 7086 to ./logs\44k\G_148800.pth 2023-09-05 08:17:08,637 44k INFO Saving model and optimizer state at iteration 7086 to ./logs\44k\D_148800.pth 2023-09-05 08:17:09,835 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_146400.pth 2023-09-05 08:17:09,869 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_146400.pth 2023-09-05 08:17:13,802 44k INFO ====> Epoch: 7086, cost 22.14 s 2023-09-05 08:17:28,199 44k INFO ====> Epoch: 7087, cost 14.40 s 2023-09-05 08:17:42,499 44k INFO ====> Epoch: 7088, cost 14.30 s 2023-09-05 08:17:57,307 44k INFO ====> Epoch: 7089, cost 14.81 s 2023-09-05 08:18:12,140 44k INFO ====> Epoch: 7090, cost 14.83 s 2023-09-05 08:18:26,702 44k INFO ====> Epoch: 7091, cost 14.56 s 2023-09-05 08:18:41,191 44k INFO ====> Epoch: 7092, cost 14.49 s 2023-09-05 08:18:55,517 44k INFO ====> Epoch: 7093, cost 14.33 s 2023-09-05 08:19:10,144 44k INFO ====> Epoch: 7094, cost 14.63 s 2023-09-05 08:19:24,502 44k INFO ====> Epoch: 7095, cost 14.36 s 2023-09-05 08:19:28,006 44k INFO Train Epoch: 7096 [19%] 2023-09-05 08:19:28,006 44k INFO Losses: [2.131225109100342, 2.7244467735290527, 10.815321922302246, 16.732831954956055, 0.6363215446472168], step: 149000, lr: 4.1191822419070954e-05, reference_loss: 33.04014587402344 2023-09-05 08:19:39,257 44k INFO ====> Epoch: 7096, cost 14.76 s 2023-09-05 08:19:53,862 44k INFO ====> Epoch: 7097, cost 14.60 s 2023-09-05 08:20:08,491 44k INFO ====> Epoch: 7098, cost 14.63 s 2023-09-05 08:20:23,023 44k INFO ====> Epoch: 7099, cost 14.53 s 2023-09-05 08:20:37,638 44k INFO ====> Epoch: 7100, cost 14.62 s 2023-09-05 08:20:52,100 44k INFO ====> Epoch: 7101, cost 14.46 s 2023-09-05 08:21:06,678 44k INFO ====> Epoch: 7102, cost 14.58 s 2023-09-05 08:21:21,191 44k INFO ====> Epoch: 7103, cost 14.51 s 2023-09-05 08:21:35,712 44k INFO ====> Epoch: 7104, cost 14.52 s 2023-09-05 08:21:46,915 44k INFO Train Epoch: 7105 [71%] 2023-09-05 08:21:46,915 44k INFO Losses: [2.172203540802002, 2.8646745681762695, 10.150720596313477, 14.351545333862305, 0.5530881285667419], step: 149200, lr: 4.114550478249282e-05, reference_loss: 30.09223175048828 2023-09-05 08:21:50,337 44k INFO ====> Epoch: 7105, cost 14.63 s 2023-09-05 08:22:04,932 44k INFO ====> Epoch: 7106, cost 14.60 s 2023-09-05 08:22:19,395 44k INFO ====> Epoch: 7107, cost 14.46 s 2023-09-05 08:22:33,805 44k INFO ====> Epoch: 7108, cost 14.41 s 2023-09-05 08:22:48,337 44k INFO ====> Epoch: 7109, cost 14.53 s 2023-09-05 08:23:03,045 44k INFO ====> Epoch: 7110, cost 14.71 s 2023-09-05 08:23:17,407 44k INFO ====> Epoch: 7111, cost 14.36 s 2023-09-05 08:23:31,871 44k INFO ====> Epoch: 7112, cost 14.46 s 2023-09-05 08:23:46,376 44k INFO ====> Epoch: 7113, cost 14.50 s 2023-09-05 08:24:01,223 44k INFO ====> Epoch: 7114, cost 14.85 s 2023-09-05 08:24:05,563 44k INFO Train Epoch: 7115 [24%] 2023-09-05 08:24:05,564 44k INFO Losses: [2.114464044570923, 2.7172603607177734, 11.10680866241455, 14.45326042175293, 0.5525346994400024], step: 149400, lr: 4.109410182230637e-05, reference_loss: 30.944326400756836 2023-09-05 08:24:16,360 44k INFO ====> Epoch: 7115, cost 15.14 s 2023-09-05 08:24:30,880 44k INFO ====> Epoch: 7116, cost 14.52 s 2023-09-05 08:24:45,396 44k INFO ====> Epoch: 7117, cost 14.52 s 2023-09-05 08:25:00,013 44k INFO ====> Epoch: 7118, cost 14.62 s 2023-09-05 08:25:14,457 44k INFO ====> Epoch: 7119, cost 14.44 s 2023-09-05 08:25:28,906 44k INFO ====> Epoch: 7120, cost 14.45 s 2023-09-05 08:25:43,435 44k INFO ====> Epoch: 7121, cost 14.53 s 2023-09-05 08:25:57,993 44k INFO ====> Epoch: 7122, cost 14.56 s 2023-09-05 08:26:12,580 44k INFO ====> Epoch: 7123, cost 14.59 s 2023-09-05 08:26:24,697 44k INFO Train Epoch: 7124 [76%] 2023-09-05 08:26:24,697 44k INFO Losses: [2.005802869796753, 2.7688827514648438, 11.190401077270508, 17.484521865844727, 0.5932639241218567], step: 149600, lr: 4.10478940664478e-05, reference_loss: 34.04287338256836 2023-09-05 08:26:30,285 44k INFO Saving model and optimizer state at iteration 7124 to ./logs\44k\G_149600.pth 2023-09-05 08:26:30,842 44k INFO Saving model and optimizer state at iteration 7124 to ./logs\44k\D_149600.pth 2023-09-05 08:26:32,109 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_147200.pth 2023-09-05 08:26:32,144 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_147200.pth 2023-09-05 08:26:34,665 44k INFO ====> Epoch: 7124, cost 22.08 s 2023-09-05 08:26:49,159 44k INFO ====> Epoch: 7125, cost 14.49 s 2023-09-05 08:27:03,908 44k INFO ====> Epoch: 7126, cost 14.75 s 2023-09-05 08:27:18,423 44k INFO ====> Epoch: 7127, cost 14.52 s 2023-09-05 08:27:32,741 44k INFO ====> Epoch: 7128, cost 14.32 s 2023-09-05 08:27:47,237 44k INFO ====> Epoch: 7129, cost 14.50 s 2023-09-05 08:28:01,843 44k INFO ====> Epoch: 7130, cost 14.61 s 2023-09-05 08:28:16,379 44k INFO ====> Epoch: 7131, cost 14.54 s 2023-09-05 08:28:30,786 44k INFO ====> Epoch: 7132, cost 14.41 s 2023-09-05 08:28:45,189 44k INFO ====> Epoch: 7133, cost 14.40 s 2023-09-05 08:28:50,196 44k INFO Train Epoch: 7134 [29%] 2023-09-05 08:28:50,196 44k INFO Losses: [2.21162748336792, 2.4405875205993652, 8.47828483581543, 14.344239234924316, 0.5637660026550293], step: 149800, lr: 4.099661305104675e-05, reference_loss: 28.03850555419922 2023-09-05 08:29:00,101 44k INFO ====> Epoch: 7134, cost 14.91 s 2023-09-05 08:29:14,655 44k INFO ====> Epoch: 7135, cost 14.55 s 2023-09-05 08:29:29,212 44k INFO ====> Epoch: 7136, cost 14.56 s 2023-09-05 08:29:43,662 44k INFO ====> Epoch: 7137, cost 14.45 s 2023-09-05 08:29:58,428 44k INFO ====> Epoch: 7138, cost 14.77 s 2023-09-05 08:30:13,031 44k INFO ====> Epoch: 7139, cost 14.60 s 2023-09-05 08:30:27,522 44k INFO ====> Epoch: 7140, cost 14.49 s 2023-09-05 08:30:42,119 44k INFO ====> Epoch: 7141, cost 14.60 s 2023-09-05 08:30:56,817 44k INFO ====> Epoch: 7142, cost 14.70 s 2023-09-05 08:31:09,648 44k INFO Train Epoch: 7143 [81%] 2023-09-05 08:31:09,648 44k INFO Losses: [2.3834612369537354, 2.3157551288604736, 9.955809593200684, 14.71825885772705, 0.5651848912239075], step: 150000, lr: 4.09505149152344e-05, reference_loss: 29.93846893310547 2023-09-05 08:31:11,817 44k INFO ====> Epoch: 7143, cost 15.00 s 2023-09-05 08:31:26,210 44k INFO ====> Epoch: 7144, cost 14.39 s 2023-09-05 08:31:40,672 44k INFO ====> Epoch: 7145, cost 14.46 s 2023-09-05 08:31:55,161 44k INFO ====> Epoch: 7146, cost 14.49 s 2023-09-05 08:32:09,957 44k INFO ====> Epoch: 7147, cost 14.80 s 2023-09-05 08:32:24,482 44k INFO ====> Epoch: 7148, cost 14.52 s 2023-09-05 08:32:39,141 44k INFO ====> Epoch: 7149, cost 14.66 s 2023-09-05 08:32:53,626 44k INFO ====> Epoch: 7150, cost 14.48 s 2023-09-05 08:33:08,344 44k INFO ====> Epoch: 7151, cost 14.72 s 2023-09-05 08:33:22,834 44k INFO ====> Epoch: 7152, cost 14.49 s 2023-09-05 08:33:28,490 44k INFO Train Epoch: 7153 [33%] 2023-09-05 08:33:28,491 44k INFO Losses: [2.0394363403320312, 2.9723453521728516, 11.205256462097168, 16.48879623413086, 0.5705010294914246], step: 150200, lr: 4.089935555532546e-05, reference_loss: 33.27633285522461 2023-09-05 08:33:37,528 44k INFO ====> Epoch: 7153, cost 14.69 s 2023-09-05 08:33:52,270 44k INFO ====> Epoch: 7154, cost 14.74 s 2023-09-05 08:34:06,877 44k INFO ====> Epoch: 7155, cost 14.61 s 2023-09-05 08:34:21,398 44k INFO ====> Epoch: 7156, cost 14.52 s 2023-09-05 08:34:35,755 44k INFO ====> Epoch: 7157, cost 14.36 s 2023-09-05 08:34:50,314 44k INFO ====> Epoch: 7158, cost 14.56 s 2023-09-05 08:35:04,812 44k INFO ====> Epoch: 7159, cost 14.50 s 2023-09-05 08:35:19,183 44k INFO ====> Epoch: 7160, cost 14.37 s 2023-09-05 08:35:33,693 44k INFO ====> Epoch: 7161, cost 14.51 s 2023-09-05 08:35:47,203 44k INFO Train Epoch: 7162 [86%] 2023-09-05 08:35:47,203 44k INFO Losses: [2.271244525909424, 2.559955596923828, 9.25683307647705, 15.467260360717773, 0.5708637833595276], step: 150400, lr: 4.085336677950441e-05, reference_loss: 30.126157760620117 2023-09-05 08:35:52,713 44k INFO Saving model and optimizer state at iteration 7162 to ./logs\44k\G_150400.pth 2023-09-05 08:35:54,047 44k INFO Saving model and optimizer state at iteration 7162 to ./logs\44k\D_150400.pth 2023-09-05 08:35:55,122 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_148000.pth 2023-09-05 08:35:55,158 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_148000.pth 2023-09-05 08:35:56,328 44k INFO ====> Epoch: 7162, cost 22.63 s 2023-09-05 08:36:10,977 44k INFO ====> Epoch: 7163, cost 14.65 s 2023-09-05 08:36:25,373 44k INFO ====> Epoch: 7164, cost 14.40 s 2023-09-05 08:36:40,020 44k INFO ====> Epoch: 7165, cost 14.65 s 2023-09-05 08:36:54,432 44k INFO ====> Epoch: 7166, cost 14.41 s 2023-09-05 08:37:09,076 44k INFO ====> Epoch: 7167, cost 14.64 s 2023-09-05 08:37:23,589 44k INFO ====> Epoch: 7168, cost 14.51 s 2023-09-05 08:37:38,057 44k INFO ====> Epoch: 7169, cost 14.47 s 2023-09-05 08:37:52,818 44k INFO ====> Epoch: 7170, cost 14.76 s 2023-09-05 08:38:07,616 44k INFO ====> Epoch: 7171, cost 14.80 s 2023-09-05 08:38:13,937 44k INFO Train Epoch: 7172 [38%] 2023-09-05 08:38:13,938 44k INFO Losses: [2.283170461654663, 2.500847101211548, 10.696866035461426, 15.968036651611328, 0.565801203250885], step: 150600, lr: 4.080232878648062e-05, reference_loss: 32.01472091674805 2023-09-05 08:38:22,296 44k INFO ====> Epoch: 7172, cost 14.68 s 2023-09-05 08:38:36,758 44k INFO ====> Epoch: 7173, cost 14.46 s 2023-09-05 08:38:51,328 44k INFO ====> Epoch: 7174, cost 14.57 s 2023-09-05 08:39:05,942 44k INFO ====> Epoch: 7175, cost 14.61 s 2023-09-05 08:39:20,283 44k INFO ====> Epoch: 7176, cost 14.34 s 2023-09-05 08:39:34,785 44k INFO ====> Epoch: 7177, cost 14.50 s 2023-09-05 08:39:49,288 44k INFO ====> Epoch: 7178, cost 14.50 s 2023-09-05 08:40:03,914 44k INFO ====> Epoch: 7179, cost 14.63 s 2023-09-05 08:40:18,403 44k INFO ====> Epoch: 7180, cost 14.49 s 2023-09-05 08:40:32,356 44k INFO Train Epoch: 7181 [90%] 2023-09-05 08:40:32,357 44k INFO Losses: [2.304758310317993, 2.4534709453582764, 8.895252227783203, 15.732059478759766, 0.49023082852363586], step: 150800, lr: 4.075644911121288e-05, reference_loss: 29.875770568847656 2023-09-05 08:40:33,043 44k INFO ====> Epoch: 7181, cost 14.64 s 2023-09-05 08:40:47,443 44k INFO ====> Epoch: 7182, cost 14.40 s 2023-09-05 08:41:02,118 44k INFO ====> Epoch: 7183, cost 14.68 s 2023-09-05 08:41:16,624 44k INFO ====> Epoch: 7184, cost 14.51 s 2023-09-05 08:41:30,921 44k INFO ====> Epoch: 7185, cost 14.30 s 2023-09-05 08:41:45,248 44k INFO ====> Epoch: 7186, cost 14.33 s 2023-09-05 08:42:00,159 44k INFO ====> Epoch: 7187, cost 14.91 s 2023-09-05 08:42:14,848 44k INFO ====> Epoch: 7188, cost 14.69 s 2023-09-05 08:42:29,206 44k INFO ====> Epoch: 7189, cost 14.36 s 2023-09-05 08:42:43,773 44k INFO ====> Epoch: 7190, cost 14.57 s 2023-09-05 08:42:50,869 44k INFO Train Epoch: 7191 [43%] 2023-09-05 08:42:50,870 44k INFO Losses: [2.1821718215942383, 2.650557041168213, 10.012654304504395, 15.567434310913086, 0.4939791262149811], step: 151000, lr: 4.070553219715192e-05, reference_loss: 30.906795501708984 2023-09-05 08:42:58,733 44k INFO ====> Epoch: 7191, cost 14.96 s 2023-09-05 08:43:13,378 44k INFO ====> Epoch: 7192, cost 14.65 s 2023-09-05 08:43:27,851 44k INFO ====> Epoch: 7193, cost 14.47 s 2023-09-05 08:43:42,412 44k INFO ====> Epoch: 7194, cost 14.56 s 2023-09-05 08:43:57,124 44k INFO ====> Epoch: 7195, cost 14.71 s 2023-09-05 08:44:11,979 44k INFO ====> Epoch: 7196, cost 14.86 s 2023-09-05 08:44:26,377 44k INFO ====> Epoch: 7197, cost 14.40 s 2023-09-05 08:44:40,955 44k INFO ====> Epoch: 7198, cost 14.58 s 2023-09-05 08:44:55,623 44k INFO ====> Epoch: 7199, cost 14.67 s 2023-09-05 08:45:10,208 44k INFO Train Epoch: 7200 [95%] 2023-09-05 08:45:10,209 44k INFO Losses: [2.0619144439697266, 2.7726874351501465, 11.433571815490723, 15.854169845581055, 0.28045228123664856], step: 151200, lr: 4.065976136361498e-05, reference_loss: 32.40279769897461 2023-09-05 08:45:15,688 44k INFO Saving model and optimizer state at iteration 7200 to ./logs\44k\G_151200.pth 2023-09-05 08:45:16,303 44k INFO Saving model and optimizer state at iteration 7200 to ./logs\44k\D_151200.pth 2023-09-05 08:45:17,560 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_148800.pth 2023-09-05 08:45:17,596 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_148800.pth 2023-09-05 08:45:17,596 44k INFO ====> Epoch: 7200, cost 21.97 s 2023-09-05 08:45:31,995 44k INFO ====> Epoch: 7201, cost 14.40 s 2023-09-05 08:45:46,500 44k INFO ====> Epoch: 7202, cost 14.51 s 2023-09-05 08:46:01,201 44k INFO ====> Epoch: 7203, cost 14.70 s 2023-09-05 08:46:15,749 44k INFO ====> Epoch: 7204, cost 14.55 s 2023-09-05 08:46:30,198 44k INFO ====> Epoch: 7205, cost 14.45 s 2023-09-05 08:46:44,654 44k INFO ====> Epoch: 7206, cost 14.46 s 2023-09-05 08:46:59,236 44k INFO ====> Epoch: 7207, cost 14.58 s 2023-09-05 08:47:13,838 44k INFO ====> Epoch: 7208, cost 14.60 s 2023-09-05 08:47:28,278 44k INFO ====> Epoch: 7209, cost 14.44 s 2023-09-05 08:47:36,013 44k INFO Train Epoch: 7210 [48%] 2023-09-05 08:47:36,014 44k INFO Losses: [2.194488763809204, 2.737534284591675, 9.211774826049805, 15.96759033203125, 0.6838672161102295], step: 151400, lr: 4.0608965241277614e-05, reference_loss: 30.795255661010742 2023-09-05 08:47:43,107 44k INFO ====> Epoch: 7210, cost 14.83 s 2023-09-05 08:47:57,912 44k INFO ====> Epoch: 7211, cost 14.80 s 2023-09-05 08:48:12,625 44k INFO ====> Epoch: 7212, cost 14.71 s 2023-09-05 08:48:27,276 44k INFO ====> Epoch: 7213, cost 14.65 s 2023-09-05 08:48:41,925 44k INFO ====> Epoch: 7214, cost 14.65 s 2023-09-05 08:48:56,654 44k INFO ====> Epoch: 7215, cost 14.73 s 2023-09-05 08:49:11,462 44k INFO ====> Epoch: 7216, cost 14.81 s 2023-09-05 08:49:25,848 44k INFO ====> Epoch: 7217, cost 14.39 s 2023-09-05 08:49:40,144 44k INFO ====> Epoch: 7218, cost 14.30 s 2023-09-05 08:49:54,829 44k INFO ====> Epoch: 7219, cost 14.68 s 2023-09-05 08:49:55,525 44k INFO Train Epoch: 7220 [0%] 2023-09-05 08:49:55,525 44k INFO Losses: [2.3276138305664062, 2.765979528427124, 10.176396369934082, 14.243830680847168, 0.596686065196991], step: 151600, lr: 4.055823257838905e-05, reference_loss: 30.110506057739258 2023-09-05 08:50:09,734 44k INFO ====> Epoch: 7220, cost 14.90 s 2023-09-05 08:50:24,324 44k INFO ====> Epoch: 7221, cost 14.59 s 2023-09-05 08:50:38,848 44k INFO ====> Epoch: 7222, cost 14.52 s 2023-09-05 08:50:53,335 44k INFO ====> Epoch: 7223, cost 14.49 s 2023-09-05 08:51:08,064 44k INFO ====> Epoch: 7224, cost 14.73 s 2023-09-05 08:51:22,437 44k INFO ====> Epoch: 7225, cost 14.37 s 2023-09-05 08:51:36,871 44k INFO ====> Epoch: 7226, cost 14.43 s 2023-09-05 08:51:51,580 44k INFO ====> Epoch: 7227, cost 14.71 s 2023-09-05 08:52:06,403 44k INFO ====> Epoch: 7228, cost 14.82 s 2023-09-05 08:52:14,960 44k INFO Train Epoch: 7229 [52%] 2023-09-05 08:52:14,960 44k INFO Losses: [1.9987976551055908, 2.9789113998413086, 10.092110633850098, 16.29352569580078, 0.4775601029396057], step: 151800, lr: 4.051262737409133e-05, reference_loss: 31.840906143188477 2023-09-05 08:52:21,328 44k INFO ====> Epoch: 7229, cost 14.92 s 2023-09-05 08:52:35,606 44k INFO ====> Epoch: 7230, cost 14.28 s 2023-09-05 08:52:50,399 44k INFO ====> Epoch: 7231, cost 14.79 s 2023-09-05 08:53:05,136 44k INFO ====> Epoch: 7232, cost 14.74 s 2023-09-05 08:53:19,614 44k INFO ====> Epoch: 7233, cost 14.48 s 2023-09-05 08:53:33,859 44k INFO ====> Epoch: 7234, cost 14.25 s 2023-09-05 08:53:48,405 44k INFO ====> Epoch: 7235, cost 14.55 s 2023-09-05 08:54:03,287 44k INFO ====> Epoch: 7236, cost 14.88 s 2023-09-05 08:54:17,740 44k INFO ====> Epoch: 7237, cost 14.45 s 2023-09-05 08:54:32,265 44k INFO ====> Epoch: 7238, cost 14.53 s 2023-09-05 08:54:33,677 44k INFO Train Epoch: 7239 [5%] 2023-09-05 08:54:33,678 44k INFO Losses: [1.9753133058547974, 3.2666237354278564, 10.810142517089844, 15.4938383102417, 0.5949196815490723], step: 152000, lr: 4.046201506582176e-05, reference_loss: 32.140838623046875 2023-09-05 08:54:39,172 44k INFO Saving model and optimizer state at iteration 7239 to ./logs\44k\G_152000.pth 2023-09-05 08:54:39,867 44k INFO Saving model and optimizer state at iteration 7239 to ./logs\44k\D_152000.pth 2023-09-05 08:54:40,945 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_149600.pth 2023-09-05 08:54:40,980 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_149600.pth 2023-09-05 08:54:54,138 44k INFO ====> Epoch: 7239, cost 21.87 s 2023-09-05 08:55:08,631 44k INFO ====> Epoch: 7240, cost 14.49 s 2023-09-05 08:55:22,950 44k INFO ====> Epoch: 7241, cost 14.32 s 2023-09-05 08:55:37,414 44k INFO ====> Epoch: 7242, cost 14.46 s 2023-09-05 08:55:52,009 44k INFO ====> Epoch: 7243, cost 14.59 s 2023-09-05 08:56:06,816 44k INFO ====> Epoch: 7244, cost 14.81 s 2023-09-05 08:56:21,203 44k INFO ====> Epoch: 7245, cost 14.39 s 2023-09-05 08:56:35,451 44k INFO ====> Epoch: 7246, cost 14.25 s 2023-09-05 08:56:50,013 44k INFO ====> Epoch: 7247, cost 14.56 s 2023-09-05 08:56:59,307 44k INFO Train Epoch: 7248 [57%] 2023-09-05 08:56:59,308 44k INFO Losses: [2.0324268341064453, 2.7925784587860107, 8.002945899963379, 13.646441459655762, 0.5743882060050964], step: 152200, lr: 4.041651805211913e-05, reference_loss: 27.04878044128418 2023-09-05 08:57:04,943 44k INFO ====> Epoch: 7248, cost 14.93 s 2023-09-05 08:57:19,349 44k INFO ====> Epoch: 7249, cost 14.41 s 2023-09-05 08:57:34,068 44k INFO ====> Epoch: 7250, cost 14.72 s 2023-09-05 08:57:48,315 44k INFO ====> Epoch: 7251, cost 14.25 s 2023-09-05 08:58:02,869 44k INFO ====> Epoch: 7252, cost 14.55 s 2023-09-05 08:58:17,314 44k INFO ====> Epoch: 7253, cost 14.44 s 2023-09-05 08:58:31,925 44k INFO ====> Epoch: 7254, cost 14.61 s 2023-09-05 08:58:46,459 44k INFO ====> Epoch: 7255, cost 14.53 s 2023-09-05 08:59:00,981 44k INFO ====> Epoch: 7256, cost 14.52 s 2023-09-05 08:59:15,521 44k INFO ====> Epoch: 7257, cost 14.54 s 2023-09-05 08:59:17,707 44k INFO Train Epoch: 7258 [10%] 2023-09-05 08:59:17,708 44k INFO Losses: [2.3437986373901367, 2.503889322280884, 7.742262840270996, 14.725687980651855, 0.5365793704986572], step: 152400, lr: 4.0366025812947665e-05, reference_loss: 27.852218627929688 2023-09-05 08:59:30,311 44k INFO ====> Epoch: 7258, cost 14.79 s 2023-09-05 08:59:44,900 44k INFO ====> Epoch: 7259, cost 14.59 s 2023-09-05 08:59:59,599 44k INFO ====> Epoch: 7260, cost 14.70 s 2023-09-05 09:00:14,283 44k INFO ====> Epoch: 7261, cost 14.68 s 2023-09-05 09:00:28,712 44k INFO ====> Epoch: 7262, cost 14.43 s 2023-09-05 09:00:43,120 44k INFO ====> Epoch: 7263, cost 14.41 s 2023-09-05 09:00:57,770 44k INFO ====> Epoch: 7264, cost 14.65 s 2023-09-05 09:01:12,353 44k INFO ====> Epoch: 7265, cost 14.58 s 2023-09-05 09:01:26,844 44k INFO ====> Epoch: 7266, cost 14.49 s 2023-09-05 09:01:36,583 44k INFO Train Epoch: 7267 [62%] 2023-09-05 09:01:36,584 44k INFO Losses: [2.227950096130371, 2.8661723136901855, 10.218622207641602, 15.007736206054688, 0.49623551964759827], step: 152600, lr: 4.032063673317629e-05, reference_loss: 30.816715240478516 2023-09-05 09:01:41,467 44k INFO ====> Epoch: 7267, cost 14.62 s 2023-09-05 09:01:56,056 44k INFO ====> Epoch: 7268, cost 14.59 s 2023-09-05 09:02:10,764 44k INFO ====> Epoch: 7269, cost 14.71 s 2023-09-05 09:02:25,270 44k INFO ====> Epoch: 7270, cost 14.51 s 2023-09-05 09:02:39,695 44k INFO ====> Epoch: 7271, cost 14.42 s 2023-09-05 09:02:54,264 44k INFO ====> Epoch: 7272, cost 14.57 s 2023-09-05 09:03:08,767 44k INFO ====> Epoch: 7273, cost 14.50 s 2023-09-05 09:03:23,024 44k INFO ====> Epoch: 7274, cost 14.26 s 2023-09-05 09:03:37,575 44k INFO ====> Epoch: 7275, cost 14.55 s 2023-09-05 09:03:52,003 44k INFO ====> Epoch: 7276, cost 14.43 s 2023-09-05 09:03:54,862 44k INFO Train Epoch: 7277 [14%] 2023-09-05 09:03:54,862 44k INFO Losses: [1.841069221496582, 3.0866622924804688, 10.381644248962402, 15.003097534179688, 0.669121265411377], step: 152800, lr: 4.027026427825942e-05, reference_loss: 30.98159408569336 2023-09-05 09:04:00,359 44k INFO Saving model and optimizer state at iteration 7277 to ./logs\44k\G_152800.pth 2023-09-05 09:04:01,129 44k INFO Saving model and optimizer state at iteration 7277 to ./logs\44k\D_152800.pth 2023-09-05 09:04:02,312 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_150400.pth 2023-09-05 09:04:02,348 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_150400.pth 2023-09-05 09:04:14,104 44k INFO ====> Epoch: 7277, cost 22.10 s 2023-09-05 09:04:28,514 44k INFO ====> Epoch: 7278, cost 14.41 s 2023-09-05 09:04:43,199 44k INFO ====> Epoch: 7279, cost 14.69 s 2023-09-05 09:04:57,797 44k INFO ====> Epoch: 7280, cost 14.60 s 2023-09-05 09:05:12,232 44k INFO ====> Epoch: 7281, cost 14.43 s 2023-09-05 09:05:26,604 44k INFO ====> Epoch: 7282, cost 14.37 s 2023-09-05 09:05:41,186 44k INFO ====> Epoch: 7283, cost 14.58 s 2023-09-05 09:05:55,997 44k INFO ====> Epoch: 7284, cost 14.81 s 2023-09-05 09:06:10,834 44k INFO ====> Epoch: 7285, cost 14.84 s 2023-09-05 09:06:21,413 44k INFO Train Epoch: 7286 [67%] 2023-09-05 09:06:21,413 44k INFO Losses: [2.149171829223633, 2.567215919494629, 9.650178909301758, 14.747551918029785, 0.5222118496894836], step: 153000, lr: 4.022498287636442e-05, reference_loss: 29.636329650878906 2023-09-05 09:06:25,584 44k INFO ====> Epoch: 7286, cost 14.75 s 2023-09-05 09:06:39,964 44k INFO ====> Epoch: 7287, cost 14.38 s 2023-09-05 09:06:54,634 44k INFO ====> Epoch: 7288, cost 14.67 s 2023-09-05 09:07:09,227 44k INFO ====> Epoch: 7289, cost 14.59 s 2023-09-05 09:07:23,736 44k INFO ====> Epoch: 7290, cost 14.51 s 2023-09-05 09:07:38,142 44k INFO ====> Epoch: 7291, cost 14.40 s 2023-09-05 09:07:52,815 44k INFO ====> Epoch: 7292, cost 14.67 s 2023-09-05 09:08:07,276 44k INFO ====> Epoch: 7293, cost 14.46 s 2023-09-05 09:08:21,731 44k INFO ====> Epoch: 7294, cost 14.45 s 2023-09-05 09:08:36,283 44k INFO ====> Epoch: 7295, cost 14.55 s 2023-09-05 09:08:39,790 44k INFO Train Epoch: 7296 [19%] 2023-09-05 09:08:39,790 44k INFO Losses: [2.2050044536590576, 2.654069423675537, 8.589664459228516, 15.084150314331055, 0.4885914921760559], step: 153200, lr: 4.0174729921534367e-05, reference_loss: 29.021480560302734 2023-09-05 09:08:51,082 44k INFO ====> Epoch: 7296, cost 14.80 s 2023-09-05 09:09:05,597 44k INFO ====> Epoch: 7297, cost 14.51 s 2023-09-05 09:09:20,147 44k INFO ====> Epoch: 7298, cost 14.55 s 2023-09-05 09:09:34,568 44k INFO ====> Epoch: 7299, cost 14.42 s 2023-09-05 09:09:48,997 44k INFO ====> Epoch: 7300, cost 14.43 s 2023-09-05 09:10:03,424 44k INFO ====> Epoch: 7301, cost 14.43 s 2023-09-05 09:10:17,913 44k INFO ====> Epoch: 7302, cost 14.49 s 2023-09-05 09:10:32,587 44k INFO ====> Epoch: 7303, cost 14.67 s 2023-09-05 09:10:47,282 44k INFO ====> Epoch: 7304, cost 14.70 s 2023-09-05 09:10:58,788 44k INFO Train Epoch: 7305 [71%] 2023-09-05 09:10:58,789 44k INFO Losses: [2.02229380607605, 2.961503744125366, 9.943975448608398, 13.88655948638916, 0.3831702470779419], step: 153400, lr: 4.012955594206828e-05, reference_loss: 29.19750213623047 2023-09-05 09:11:02,308 44k INFO ====> Epoch: 7305, cost 15.03 s 2023-09-05 09:11:16,804 44k INFO ====> Epoch: 7306, cost 14.50 s 2023-09-05 09:11:31,207 44k INFO ====> Epoch: 7307, cost 14.40 s 2023-09-05 09:11:45,729 44k INFO ====> Epoch: 7308, cost 14.52 s 2023-09-05 09:12:00,421 44k INFO ====> Epoch: 7309, cost 14.69 s 2023-09-05 09:12:15,111 44k INFO ====> Epoch: 7310, cost 14.69 s 2023-09-05 09:12:29,806 44k INFO ====> Epoch: 7311, cost 14.70 s 2023-09-05 09:12:44,292 44k INFO ====> Epoch: 7312, cost 14.49 s 2023-09-05 09:12:58,880 44k INFO ====> Epoch: 7313, cost 14.59 s 2023-09-05 09:13:13,487 44k INFO ====> Epoch: 7314, cost 14.61 s 2023-09-05 09:13:17,702 44k INFO Train Epoch: 7315 [24%] 2023-09-05 09:13:17,702 44k INFO Losses: [2.2444260120391846, 2.494581460952759, 9.73157024383545, 15.126556396484375, 0.6820967197418213], step: 153600, lr: 4.007942220383139e-05, reference_loss: 30.27923011779785 2023-09-05 09:13:23,230 44k INFO Saving model and optimizer state at iteration 7315 to ./logs\44k\G_153600.pth 2023-09-05 09:13:23,961 44k INFO Saving model and optimizer state at iteration 7315 to ./logs\44k\D_153600.pth 2023-09-05 09:13:25,080 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_151200.pth 2023-09-05 09:13:25,116 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_151200.pth 2023-09-05 09:13:35,268 44k INFO ====> Epoch: 7315, cost 21.78 s 2023-09-05 09:13:49,803 44k INFO ====> Epoch: 7316, cost 14.53 s 2023-09-05 09:14:04,499 44k INFO ====> Epoch: 7317, cost 14.70 s 2023-09-05 09:14:19,077 44k INFO ====> Epoch: 7318, cost 14.58 s 2023-09-05 09:14:33,605 44k INFO ====> Epoch: 7319, cost 14.53 s 2023-09-05 09:14:48,026 44k INFO ====> Epoch: 7320, cost 14.42 s 2023-09-05 09:15:02,703 44k INFO ====> Epoch: 7321, cost 14.68 s 2023-09-05 09:15:17,134 44k INFO ====> Epoch: 7322, cost 14.43 s 2023-09-05 09:15:31,351 44k INFO ====> Epoch: 7323, cost 14.22 s 2023-09-05 09:15:43,272 44k INFO Train Epoch: 7324 [76%] 2023-09-05 09:15:43,272 44k INFO Losses: [2.206782102584839, 2.510969400405884, 9.926947593688965, 16.899757385253906, 0.7275641560554504], step: 153800, lr: 4.0034355391952754e-05, reference_loss: 32.27202224731445 2023-09-05 09:15:46,113 44k INFO ====> Epoch: 7324, cost 14.76 s 2023-09-05 09:16:00,706 44k INFO ====> Epoch: 7325, cost 14.59 s 2023-09-05 09:16:15,188 44k INFO ====> Epoch: 7326, cost 14.48 s 2023-09-05 09:16:29,612 44k INFO ====> Epoch: 7327, cost 14.42 s 2023-09-05 09:16:44,113 44k INFO ====> Epoch: 7328, cost 14.50 s 2023-09-05 09:16:58,785 44k INFO ====> Epoch: 7329, cost 14.67 s 2023-09-05 09:17:13,410 44k INFO ====> Epoch: 7330, cost 14.62 s 2023-09-05 09:17:27,786 44k INFO ====> Epoch: 7331, cost 14.38 s 2023-09-05 09:17:42,337 44k INFO ====> Epoch: 7332, cost 14.55 s 2023-09-05 09:17:56,987 44k INFO ====> Epoch: 7333, cost 14.65 s 2023-09-05 09:18:02,088 44k INFO Train Epoch: 7334 [29%] 2023-09-05 09:18:02,088 44k INFO Losses: [2.122210741043091, 2.689253330230713, 8.8027925491333, 15.1478910446167, 0.5344655513763428], step: 154000, lr: 3.998434058748793e-05, reference_loss: 29.296613693237305 2023-09-05 09:18:12,248 44k INFO ====> Epoch: 7334, cost 15.26 s 2023-09-05 09:18:26,707 44k INFO ====> Epoch: 7335, cost 14.46 s 2023-09-05 09:18:41,151 44k INFO ====> Epoch: 7336, cost 14.44 s 2023-09-05 09:18:55,977 44k INFO ====> Epoch: 7337, cost 14.83 s 2023-09-05 09:19:10,432 44k INFO ====> Epoch: 7338, cost 14.46 s 2023-09-05 09:19:24,761 44k INFO ====> Epoch: 7339, cost 14.33 s 2023-09-05 09:19:39,283 44k INFO ====> Epoch: 7340, cost 14.52 s 2023-09-05 09:19:53,996 44k INFO ====> Epoch: 7341, cost 14.71 s 2023-09-05 09:20:08,693 44k INFO ====> Epoch: 7342, cost 14.70 s 2023-09-05 09:20:21,422 44k INFO Train Epoch: 7343 [81%] 2023-09-05 09:20:21,422 44k INFO Losses: [2.0944299697875977, 2.5546977519989014, 9.773272514343262, 14.781725883483887, 0.6688197255134583], step: 154200, lr: 3.993938068895987e-05, reference_loss: 29.87294578552246 2023-09-05 09:20:23,474 44k INFO ====> Epoch: 7343, cost 14.78 s 2023-09-05 09:20:37,825 44k INFO ====> Epoch: 7344, cost 14.35 s 2023-09-05 09:20:52,166 44k INFO ====> Epoch: 7345, cost 14.34 s 2023-09-05 09:21:06,567 44k INFO ====> Epoch: 7346, cost 14.40 s 2023-09-05 09:21:20,944 44k INFO ====> Epoch: 7347, cost 14.38 s 2023-09-05 09:21:35,239 44k INFO ====> Epoch: 7348, cost 14.30 s 2023-09-05 09:21:49,614 44k INFO ====> Epoch: 7349, cost 14.38 s 2023-09-05 09:22:04,328 44k INFO ====> Epoch: 7350, cost 14.71 s 2023-09-05 09:22:18,922 44k INFO ====> Epoch: 7351, cost 14.59 s 2023-09-05 09:22:33,507 44k INFO ====> Epoch: 7352, cost 14.58 s 2023-09-05 09:22:39,047 44k INFO Train Epoch: 7353 [33%] 2023-09-05 09:22:39,048 44k INFO Losses: [2.3118979930877686, 2.7044100761413574, 7.640371322631836, 14.25161361694336, 0.5984688401222229], step: 154400, lr: 3.988948453611696e-05, reference_loss: 27.50676155090332 2023-09-05 09:22:44,661 44k INFO Saving model and optimizer state at iteration 7353 to ./logs\44k\G_154400.pth 2023-09-05 09:22:45,243 44k INFO Saving model and optimizer state at iteration 7353 to ./logs\44k\D_154400.pth 2023-09-05 09:22:46,897 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_152000.pth 2023-09-05 09:22:46,935 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_152000.pth 2023-09-05 09:22:56,051 44k INFO ====> Epoch: 7353, cost 22.54 s 2023-09-05 09:23:10,689 44k INFO ====> Epoch: 7354, cost 14.64 s 2023-09-05 09:23:25,129 44k INFO ====> Epoch: 7355, cost 14.44 s 2023-09-05 09:23:39,496 44k INFO ====> Epoch: 7356, cost 14.37 s 2023-09-05 09:23:54,320 44k INFO ====> Epoch: 7357, cost 14.82 s 2023-09-05 09:24:09,060 44k INFO ====> Epoch: 7358, cost 14.74 s 2023-09-05 09:24:23,504 44k INFO ====> Epoch: 7359, cost 14.44 s 2023-09-05 09:24:38,065 44k INFO ====> Epoch: 7360, cost 14.56 s 2023-09-05 09:24:52,733 44k INFO ====> Epoch: 7361, cost 14.67 s 2023-09-05 09:25:06,243 44k INFO Train Epoch: 7362 [86%] 2023-09-05 09:25:06,243 44k INFO Losses: [1.8825876712799072, 3.022444725036621, 10.694206237792969, 15.654741287231445, 0.684009850025177], step: 154600, lr: 3.984463129730573e-05, reference_loss: 31.93798828125 2023-09-05 09:25:07,631 44k INFO ====> Epoch: 7362, cost 14.90 s 2023-09-05 09:25:22,004 44k INFO ====> Epoch: 7363, cost 14.37 s 2023-09-05 09:25:36,362 44k INFO ====> Epoch: 7364, cost 14.36 s 2023-09-05 09:25:51,006 44k INFO ====> Epoch: 7365, cost 14.64 s 2023-09-05 09:26:05,692 44k INFO ====> Epoch: 7366, cost 14.69 s 2023-09-05 09:26:20,315 44k INFO ====> Epoch: 7367, cost 14.62 s 2023-09-05 09:26:34,692 44k INFO ====> Epoch: 7368, cost 14.38 s 2023-09-05 09:26:49,231 44k INFO ====> Epoch: 7369, cost 14.54 s 2023-09-05 09:27:04,017 44k INFO ====> Epoch: 7370, cost 14.79 s 2023-09-05 09:27:18,552 44k INFO ====> Epoch: 7371, cost 14.54 s 2023-09-05 09:27:24,735 44k INFO Train Epoch: 7372 [38%] 2023-09-05 09:27:24,735 44k INFO Losses: [2.072946310043335, 2.798567771911621, 11.319185256958008, 15.325465202331543, 0.5492734909057617], step: 154800, lr: 3.979485351460393e-05, reference_loss: 32.0654411315918 2023-09-05 09:27:33,205 44k INFO ====> Epoch: 7372, cost 14.65 s 2023-09-05 09:27:47,760 44k INFO ====> Epoch: 7373, cost 14.56 s 2023-09-05 09:28:02,349 44k INFO ====> Epoch: 7374, cost 14.59 s 2023-09-05 09:28:16,984 44k INFO ====> Epoch: 7375, cost 14.64 s 2023-09-05 09:28:31,268 44k INFO ====> Epoch: 7376, cost 14.28 s 2023-09-05 09:28:45,810 44k INFO ====> Epoch: 7377, cost 14.54 s 2023-09-05 09:29:00,675 44k INFO ====> Epoch: 7378, cost 14.87 s 2023-09-05 09:29:15,416 44k INFO ====> Epoch: 7379, cost 14.74 s 2023-09-05 09:29:29,768 44k INFO ====> Epoch: 7380, cost 14.35 s 2023-09-05 09:29:43,800 44k INFO Train Epoch: 7381 [90%] 2023-09-05 09:29:43,800 44k INFO Losses: [2.2481117248535156, 2.7109875679016113, 9.87670612335205, 16.886438369750977, 0.4215955138206482], step: 155000, lr: 3.9750106682477465e-05, reference_loss: 32.14384078979492 2023-09-05 09:29:44,483 44k INFO ====> Epoch: 7381, cost 14.72 s 2023-09-05 09:29:59,290 44k INFO ====> Epoch: 7382, cost 14.81 s 2023-09-05 09:30:14,021 44k INFO ====> Epoch: 7383, cost 14.73 s 2023-09-05 09:30:28,434 44k INFO ====> Epoch: 7384, cost 14.41 s 2023-09-05 09:30:42,928 44k INFO ====> Epoch: 7385, cost 14.49 s 2023-09-05 09:30:57,623 44k INFO ====> Epoch: 7386, cost 14.70 s 2023-09-05 09:31:12,104 44k INFO ====> Epoch: 7387, cost 14.48 s 2023-09-05 09:31:26,516 44k INFO ====> Epoch: 7388, cost 14.41 s 2023-09-05 09:31:40,793 44k INFO ====> Epoch: 7389, cost 14.28 s 2023-09-05 09:31:55,480 44k INFO ====> Epoch: 7390, cost 14.69 s 2023-09-05 09:32:02,524 44k INFO Train Epoch: 7391 [43%] 2023-09-05 09:32:02,524 44k INFO Losses: [2.1203863620758057, 2.740954875946045, 10.096488952636719, 14.694271087646484, 0.6848376393318176], step: 155200, lr: 3.9700446989103706e-05, reference_loss: 30.336938858032227 2023-09-05 09:32:08,071 44k INFO Saving model and optimizer state at iteration 7391 to ./logs\44k\G_155200.pth 2023-09-05 09:32:08,730 44k INFO Saving model and optimizer state at iteration 7391 to ./logs\44k\D_155200.pth 2023-09-05 09:32:10,459 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_152800.pth 2023-09-05 09:32:10,494 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_152800.pth 2023-09-05 09:32:17,928 44k INFO ====> Epoch: 7391, cost 22.45 s 2023-09-05 09:32:32,385 44k INFO ====> Epoch: 7392, cost 14.46 s 2023-09-05 09:32:46,901 44k INFO ====> Epoch: 7393, cost 14.52 s 2023-09-05 09:33:01,487 44k INFO ====> Epoch: 7394, cost 14.59 s 2023-09-05 09:33:15,959 44k INFO ====> Epoch: 7395, cost 14.47 s 2023-09-05 09:33:30,397 44k INFO ====> Epoch: 7396, cost 14.44 s 2023-09-05 09:33:44,752 44k INFO ====> Epoch: 7397, cost 14.35 s 2023-09-05 09:33:59,397 44k INFO ====> Epoch: 7398, cost 14.65 s 2023-09-05 09:34:13,718 44k INFO ====> Epoch: 7399, cost 14.32 s 2023-09-05 09:34:28,200 44k INFO Train Epoch: 7400 [95%] 2023-09-05 09:34:28,200 44k INFO Losses: [1.9535939693450928, 2.945981502532959, 8.380454063415527, 13.335214614868164, 0.5680617690086365], step: 155400, lr: 3.9655806311230246e-05, reference_loss: 27.183305740356445 2023-09-05 09:34:28,465 44k INFO ====> Epoch: 7400, cost 14.75 s 2023-09-05 09:34:43,127 44k INFO ====> Epoch: 7401, cost 14.66 s 2023-09-05 09:34:57,888 44k INFO ====> Epoch: 7402, cost 14.76 s 2023-09-05 09:35:12,528 44k INFO ====> Epoch: 7403, cost 14.64 s 2023-09-05 09:35:26,771 44k INFO ====> Epoch: 7404, cost 14.24 s 2023-09-05 09:35:41,210 44k INFO ====> Epoch: 7405, cost 14.44 s 2023-09-05 09:35:56,062 44k INFO ====> Epoch: 7406, cost 14.85 s 2023-09-05 09:36:10,853 44k INFO ====> Epoch: 7407, cost 14.79 s 2023-09-05 09:36:25,374 44k INFO ====> Epoch: 7408, cost 14.52 s 2023-09-05 09:36:39,987 44k INFO ====> Epoch: 7409, cost 14.61 s 2023-09-05 09:36:47,790 44k INFO Train Epoch: 7410 [48%] 2023-09-05 09:36:47,791 44k INFO Losses: [2.0769290924072266, 2.8984568119049072, 11.504631042480469, 16.685144424438477, 0.5828172564506531], step: 155600, lr: 3.9606264427037714e-05, reference_loss: 33.74797821044922 2023-09-05 09:36:54,967 44k INFO ====> Epoch: 7410, cost 14.98 s 2023-09-05 09:37:09,489 44k INFO ====> Epoch: 7411, cost 14.52 s 2023-09-05 09:37:23,816 44k INFO ====> Epoch: 7412, cost 14.33 s 2023-09-05 09:37:38,274 44k INFO ====> Epoch: 7413, cost 14.46 s 2023-09-05 09:37:52,974 44k INFO ====> Epoch: 7414, cost 14.70 s 2023-09-05 09:38:07,582 44k INFO ====> Epoch: 7415, cost 14.61 s 2023-09-05 09:38:22,124 44k INFO ====> Epoch: 7416, cost 14.54 s 2023-09-05 09:38:36,691 44k INFO ====> Epoch: 7417, cost 14.57 s 2023-09-05 09:38:51,354 44k INFO ====> Epoch: 7418, cost 14.66 s 2023-09-05 09:39:05,735 44k INFO ====> Epoch: 7419, cost 14.38 s 2023-09-05 09:39:06,503 44k INFO Train Epoch: 7420 [0%] 2023-09-05 09:39:06,503 44k INFO Losses: [2.1168272495269775, 2.9296112060546875, 8.693831443786621, 13.723581314086914, 0.6629951596260071], step: 155800, lr: 3.95567844353779e-05, reference_loss: 28.126846313476562 2023-09-05 09:39:20,600 44k INFO ====> Epoch: 7420, cost 14.86 s 2023-09-05 09:39:34,975 44k INFO ====> Epoch: 7421, cost 14.37 s 2023-09-05 09:39:49,528 44k INFO ====> Epoch: 7422, cost 14.55 s 2023-09-05 09:40:04,146 44k INFO ====> Epoch: 7423, cost 14.62 s 2023-09-05 09:40:18,700 44k INFO ====> Epoch: 7424, cost 14.55 s 2023-09-05 09:40:33,214 44k INFO ====> Epoch: 7425, cost 14.51 s 2023-09-05 09:40:47,751 44k INFO ====> Epoch: 7426, cost 14.54 s 2023-09-05 09:41:02,394 44k INFO ====> Epoch: 7427, cost 14.64 s 2023-09-05 09:41:16,823 44k INFO ====> Epoch: 7428, cost 14.43 s 2023-09-05 09:41:25,263 44k INFO Train Epoch: 7429 [52%] 2023-09-05 09:41:25,264 44k INFO Losses: [1.9001107215881348, 3.106879949569702, 9.82451057434082, 14.98658561706543, 0.5849463939666748], step: 156000, lr: 3.951230529709076e-05, reference_loss: 30.403032302856445 2023-09-05 09:41:30,759 44k INFO Saving model and optimizer state at iteration 7429 to ./logs\44k\G_156000.pth 2023-09-05 09:41:31,428 44k INFO Saving model and optimizer state at iteration 7429 to ./logs\44k\D_156000.pth 2023-09-05 09:41:32,723 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_153600.pth 2023-09-05 09:41:32,757 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_153600.pth 2023-09-05 09:41:38,835 44k INFO ====> Epoch: 7429, cost 22.01 s 2023-09-05 09:41:53,636 44k INFO ====> Epoch: 7430, cost 14.80 s 2023-09-05 09:42:08,442 44k INFO ====> Epoch: 7431, cost 14.81 s 2023-09-05 09:42:23,002 44k INFO ====> Epoch: 7432, cost 14.56 s 2023-09-05 09:42:37,495 44k INFO ====> Epoch: 7433, cost 14.49 s 2023-09-05 09:42:51,997 44k INFO ====> Epoch: 7434, cost 14.50 s 2023-09-05 09:43:06,688 44k INFO ====> Epoch: 7435, cost 14.69 s 2023-09-05 09:43:21,228 44k INFO ====> Epoch: 7436, cost 14.54 s 2023-09-05 09:43:35,564 44k INFO ====> Epoch: 7437, cost 14.34 s 2023-09-05 09:43:50,044 44k INFO ====> Epoch: 7438, cost 14.48 s 2023-09-05 09:43:51,510 44k INFO Train Epoch: 7439 [5%] 2023-09-05 09:43:51,510 44k INFO Losses: [2.725321054458618, 2.0329742431640625, 6.1479105949401855, 12.255423545837402, 0.6165268421173096], step: 156200, lr: 3.946294268830037e-05, reference_loss: 23.778156280517578 2023-09-05 09:44:05,080 44k INFO ====> Epoch: 7439, cost 15.04 s 2023-09-05 09:44:19,867 44k INFO ====> Epoch: 7440, cost 14.79 s 2023-09-05 09:44:34,325 44k INFO ====> Epoch: 7441, cost 14.46 s 2023-09-05 09:44:48,783 44k INFO ====> Epoch: 7442, cost 14.46 s 2023-09-05 09:45:03,552 44k INFO ====> Epoch: 7443, cost 14.77 s 2023-09-05 09:45:18,032 44k INFO ====> Epoch: 7444, cost 14.48 s 2023-09-05 09:45:32,438 44k INFO ====> Epoch: 7445, cost 14.41 s 2023-09-05 09:45:47,136 44k INFO ====> Epoch: 7446, cost 14.70 s 2023-09-05 09:46:01,667 44k INFO ====> Epoch: 7447, cost 14.53 s 2023-09-05 09:46:10,889 44k INFO Train Epoch: 7448 [57%] 2023-09-05 09:46:10,889 44k INFO Losses: [1.9882009029388428, 2.986996650695801, 10.276860237121582, 16.179521560668945, 0.5230643153190613], step: 156400, lr: 3.94185690692081e-05, reference_loss: 31.95464324951172 2023-09-05 09:46:16,564 44k INFO ====> Epoch: 7448, cost 14.90 s 2023-09-05 09:46:30,997 44k INFO ====> Epoch: 7449, cost 14.43 s 2023-09-05 09:46:45,495 44k INFO ====> Epoch: 7450, cost 14.50 s 2023-09-05 09:47:00,114 44k INFO ====> Epoch: 7451, cost 14.62 s 2023-09-05 09:47:14,546 44k INFO ====> Epoch: 7452, cost 14.43 s 2023-09-05 09:47:28,867 44k INFO ====> Epoch: 7453, cost 14.32 s 2023-09-05 09:47:43,162 44k INFO ====> Epoch: 7454, cost 14.30 s 2023-09-05 09:47:58,019 44k INFO ====> Epoch: 7455, cost 14.86 s 2023-09-05 09:48:12,757 44k INFO ====> Epoch: 7456, cost 14.74 s 2023-09-05 09:48:27,248 44k INFO ====> Epoch: 7457, cost 14.49 s 2023-09-05 09:48:29,367 44k INFO Train Epoch: 7458 [10%] 2023-09-05 09:48:29,368 44k INFO Losses: [2.236239433288574, 2.8553152084350586, 8.77932357788086, 15.368985176086426, 0.6061642169952393], step: 156600, lr: 3.9369323564816245e-05, reference_loss: 29.84602928161621 2023-09-05 09:48:42,293 44k INFO ====> Epoch: 7458, cost 15.05 s 2023-09-05 09:48:56,940 44k INFO ====> Epoch: 7459, cost 14.65 s 2023-09-05 09:49:11,634 44k INFO ====> Epoch: 7460, cost 14.69 s 2023-09-05 09:49:25,972 44k INFO ====> Epoch: 7461, cost 14.34 s 2023-09-05 09:49:40,383 44k INFO ====> Epoch: 7462, cost 14.41 s 2023-09-05 09:49:55,150 44k INFO ====> Epoch: 7463, cost 14.77 s 2023-09-05 09:50:09,900 44k INFO ====> Epoch: 7464, cost 14.75 s 2023-09-05 09:50:24,480 44k INFO ====> Epoch: 7465, cost 14.58 s 2023-09-05 09:50:39,041 44k INFO ====> Epoch: 7466, cost 14.56 s 2023-09-05 09:50:49,070 44k INFO Train Epoch: 7467 [62%] 2023-09-05 09:50:49,071 44k INFO Losses: [2.1491751670837402, 2.5733511447906494, 7.966416835784912, 13.89090347290039, 0.5430083870887756], step: 156800, lr: 3.93250552145925e-05, reference_loss: 27.12285614013672 2023-09-05 09:50:54,563 44k INFO Saving model and optimizer state at iteration 7467 to ./logs\44k\G_156800.pth 2023-09-05 09:50:55,256 44k INFO Saving model and optimizer state at iteration 7467 to ./logs\44k\D_156800.pth 2023-09-05 09:50:56,400 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_154400.pth 2023-09-05 09:50:56,438 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_154400.pth 2023-09-05 09:51:01,155 44k INFO ====> Epoch: 7467, cost 22.11 s 2023-09-05 09:51:15,578 44k INFO ====> Epoch: 7468, cost 14.42 s 2023-09-05 09:51:29,997 44k INFO ====> Epoch: 7469, cost 14.42 s 2023-09-05 09:51:44,372 44k INFO ====> Epoch: 7470, cost 14.38 s 2023-09-05 09:51:59,187 44k INFO ====> Epoch: 7471, cost 14.81 s 2023-09-05 09:52:13,682 44k INFO ====> Epoch: 7472, cost 14.50 s 2023-09-05 09:52:28,224 44k INFO ====> Epoch: 7473, cost 14.54 s 2023-09-05 09:52:42,723 44k INFO ====> Epoch: 7474, cost 14.50 s 2023-09-05 09:52:57,150 44k INFO ====> Epoch: 7475, cost 14.43 s 2023-09-05 09:53:11,837 44k INFO ====> Epoch: 7476, cost 14.69 s 2023-09-05 09:53:14,656 44k INFO Train Epoch: 7477 [14%] 2023-09-05 09:53:14,656 44k INFO Losses: [2.3658769130706787, 2.4697680473327637, 9.634699821472168, 16.400999069213867, 0.6990527510643005], step: 157000, lr: 3.9275926536788894e-05, reference_loss: 31.570396423339844 2023-09-05 09:53:26,609 44k INFO ====> Epoch: 7477, cost 14.77 s 2023-09-05 09:53:41,005 44k INFO ====> Epoch: 7478, cost 14.40 s 2023-09-05 09:53:55,903 44k INFO ====> Epoch: 7479, cost 14.90 s 2023-09-05 09:54:10,711 44k INFO ====> Epoch: 7480, cost 14.81 s 2023-09-05 09:54:25,014 44k INFO ====> Epoch: 7481, cost 14.30 s 2023-09-05 09:54:39,545 44k INFO ====> Epoch: 7482, cost 14.53 s 2023-09-05 09:54:54,267 44k INFO ====> Epoch: 7483, cost 14.72 s 2023-09-05 09:55:08,872 44k INFO ====> Epoch: 7484, cost 14.60 s 2023-09-05 09:55:23,234 44k INFO ====> Epoch: 7485, cost 14.36 s 2023-09-05 09:55:33,551 44k INFO Train Epoch: 7486 [67%] 2023-09-05 09:55:33,551 44k INFO Losses: [2.1817917823791504, 2.917156457901001, 9.632688522338867, 15.967326164245605, 0.5238774418830872], step: 157200, lr: 3.923176320570118e-05, reference_loss: 31.222841262817383 2023-09-05 09:55:37,685 44k INFO ====> Epoch: 7486, cost 14.45 s 2023-09-05 09:55:52,206 44k INFO ====> Epoch: 7487, cost 14.52 s 2023-09-05 09:56:06,813 44k INFO ====> Epoch: 7488, cost 14.61 s 2023-09-05 09:56:21,174 44k INFO ====> Epoch: 7489, cost 14.36 s 2023-09-05 09:56:35,575 44k INFO ====> Epoch: 7490, cost 14.40 s 2023-09-05 09:56:50,072 44k INFO ====> Epoch: 7491, cost 14.50 s 2023-09-05 09:57:04,743 44k INFO ====> Epoch: 7492, cost 14.67 s 2023-09-05 09:57:19,252 44k INFO ====> Epoch: 7493, cost 14.51 s 2023-09-05 09:57:33,545 44k INFO ====> Epoch: 7494, cost 14.29 s 2023-09-05 09:57:48,380 44k INFO ====> Epoch: 7495, cost 14.83 s 2023-09-05 09:57:51,864 44k INFO Train Epoch: 7496 [19%] 2023-09-05 09:57:51,864 44k INFO Losses: [2.218684196472168, 2.4402246475219727, 9.461141586303711, 14.275413513183594, 0.5015243291854858], step: 157400, lr: 3.918275107733461e-05, reference_loss: 28.896987915039062 2023-09-05 09:58:03,157 44k INFO ====> Epoch: 7496, cost 14.78 s 2023-09-05 09:58:17,800 44k INFO ====> Epoch: 7497, cost 14.64 s 2023-09-05 09:58:32,273 44k INFO ====> Epoch: 7498, cost 14.47 s 2023-09-05 09:58:46,844 44k INFO ====> Epoch: 7499, cost 14.57 s 2023-09-05 09:59:01,632 44k INFO ====> Epoch: 7500, cost 14.79 s 2023-09-05 09:59:16,396 44k INFO ====> Epoch: 7501, cost 14.76 s 2023-09-05 09:59:30,665 44k INFO ====> Epoch: 7502, cost 14.27 s 2023-09-05 09:59:45,035 44k INFO ====> Epoch: 7503, cost 14.37 s 2023-09-05 09:59:59,954 44k INFO ====> Epoch: 7504, cost 14.92 s 2023-09-05 10:00:11,369 44k INFO Train Epoch: 7505 [71%] 2023-09-05 10:00:11,370 44k INFO Losses: [2.0793328285217285, 2.8846611976623535, 10.040884971618652, 15.163284301757812, 0.6081374883651733], step: 157600, lr: 3.913869251624286e-05, reference_loss: 30.77630043029785 2023-09-05 10:00:16,823 44k INFO Saving model and optimizer state at iteration 7505 to ./logs\44k\G_157600.pth 2023-09-05 10:00:17,440 44k INFO Saving model and optimizer state at iteration 7505 to ./logs\44k\D_157600.pth 2023-09-05 10:00:19,067 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_155200.pth 2023-09-05 10:00:19,103 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_155200.pth 2023-09-05 10:00:22,306 44k INFO ====> Epoch: 7505, cost 22.35 s 2023-09-05 10:00:36,843 44k INFO ====> Epoch: 7506, cost 14.54 s 2023-09-05 10:00:51,512 44k INFO ====> Epoch: 7507, cost 14.67 s 2023-09-05 10:01:06,161 44k INFO ====> Epoch: 7508, cost 14.65 s 2023-09-05 10:01:20,610 44k INFO ====> Epoch: 7509, cost 14.45 s 2023-09-05 10:01:34,949 44k INFO ====> Epoch: 7510, cost 14.34 s 2023-09-05 10:01:49,607 44k INFO ====> Epoch: 7511, cost 14.66 s 2023-09-05 10:02:04,305 44k INFO ====> Epoch: 7512, cost 14.70 s 2023-09-05 10:02:18,737 44k INFO ====> Epoch: 7513, cost 14.43 s 2023-09-05 10:02:33,147 44k INFO ====> Epoch: 7514, cost 14.41 s 2023-09-05 10:02:37,339 44k INFO Train Epoch: 7515 [24%] 2023-09-05 10:02:37,340 44k INFO Losses: [2.353304624557495, 2.4637436866760254, 9.164432525634766, 14.931050300598145, 0.5702353715896606], step: 157800, lr: 3.908979666081959e-05, reference_loss: 29.48276710510254 2023-09-05 10:02:47,927 44k INFO ====> Epoch: 7515, cost 14.78 s 2023-09-05 10:03:02,440 44k INFO ====> Epoch: 7516, cost 14.51 s 2023-09-05 10:03:16,883 44k INFO ====> Epoch: 7517, cost 14.44 s 2023-09-05 10:03:31,350 44k INFO ====> Epoch: 7518, cost 14.47 s 2023-09-05 10:03:45,601 44k INFO ====> Epoch: 7519, cost 14.25 s 2023-09-05 10:04:00,307 44k INFO ====> Epoch: 7520, cost 14.71 s 2023-09-05 10:04:14,826 44k INFO ====> Epoch: 7521, cost 14.52 s 2023-09-05 10:04:29,351 44k INFO ====> Epoch: 7522, cost 14.53 s 2023-09-05 10:04:43,805 44k INFO ====> Epoch: 7523, cost 14.45 s 2023-09-05 10:04:55,880 44k INFO Train Epoch: 7524 [76%] 2023-09-05 10:04:55,880 44k INFO Losses: [2.186365842819214, 2.674844980239868, 10.447120666503906, 16.465124130249023, 0.5122056603431702], step: 158000, lr: 3.904584262117481e-05, reference_loss: 32.28566360473633 2023-09-05 10:04:58,763 44k INFO ====> Epoch: 7524, cost 14.96 s 2023-09-05 10:05:13,303 44k INFO ====> Epoch: 7525, cost 14.54 s 2023-09-05 10:05:27,833 44k INFO ====> Epoch: 7526, cost 14.53 s 2023-09-05 10:05:42,212 44k INFO ====> Epoch: 7527, cost 14.38 s 2023-09-05 10:05:56,872 44k INFO ====> Epoch: 7528, cost 14.66 s 2023-09-05 10:06:11,553 44k INFO ====> Epoch: 7529, cost 14.68 s 2023-09-05 10:06:26,091 44k INFO ====> Epoch: 7530, cost 14.54 s 2023-09-05 10:06:40,694 44k INFO ====> Epoch: 7531, cost 14.60 s 2023-09-05 10:06:55,296 44k INFO ====> Epoch: 7532, cost 14.60 s 2023-09-05 10:07:10,059 44k INFO ====> Epoch: 7533, cost 14.76 s 2023-09-05 10:07:14,989 44k INFO Train Epoch: 7534 [29%] 2023-09-05 10:07:14,989 44k INFO Losses: [2.1121716499328613, 2.754978656768799, 8.211852073669434, 13.72228717803955, 0.4572063386440277], step: 158200, lr: 3.899706276285705e-05, reference_loss: 27.258495330810547 2023-09-05 10:07:24,773 44k INFO ====> Epoch: 7534, cost 14.71 s 2023-09-05 10:07:39,073 44k INFO ====> Epoch: 7535, cost 14.30 s 2023-09-05 10:07:53,621 44k INFO ====> Epoch: 7536, cost 14.55 s 2023-09-05 10:08:08,252 44k INFO ====> Epoch: 7537, cost 14.63 s 2023-09-05 10:08:22,888 44k INFO ====> Epoch: 7538, cost 14.64 s 2023-09-05 10:08:37,366 44k INFO ====> Epoch: 7539, cost 14.48 s 2023-09-05 10:08:52,119 44k INFO ====> Epoch: 7540, cost 14.75 s 2023-09-05 10:09:06,719 44k INFO ====> Epoch: 7541, cost 14.60 s 2023-09-05 10:09:21,244 44k INFO ====> Epoch: 7542, cost 14.53 s 2023-09-05 10:09:33,811 44k INFO Train Epoch: 7543 [81%] 2023-09-05 10:09:33,811 44k INFO Losses: [2.174574851989746, 2.7155075073242188, 9.54919719696045, 14.12016487121582, 0.5961432456970215], step: 158400, lr: 3.8953212996699865e-05, reference_loss: 29.155588150024414 2023-09-05 10:09:39,380 44k INFO Saving model and optimizer state at iteration 7543 to ./logs\44k\G_158400.pth 2023-09-05 10:09:40,016 44k INFO Saving model and optimizer state at iteration 7543 to ./logs\44k\D_158400.pth 2023-09-05 10:09:41,256 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_156000.pth 2023-09-05 10:09:41,293 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_156000.pth 2023-09-05 10:09:43,240 44k INFO ====> Epoch: 7543, cost 22.00 s 2023-09-05 10:09:57,941 44k INFO ====> Epoch: 7544, cost 14.70 s 2023-09-05 10:10:12,635 44k INFO ====> Epoch: 7545, cost 14.69 s 2023-09-05 10:10:27,103 44k INFO ====> Epoch: 7546, cost 14.47 s 2023-09-05 10:10:41,503 44k INFO ====> Epoch: 7547, cost 14.40 s 2023-09-05 10:10:56,358 44k INFO ====> Epoch: 7548, cost 14.86 s 2023-09-05 10:11:11,301 44k INFO ====> Epoch: 7549, cost 14.94 s 2023-09-05 10:11:25,512 44k INFO ====> Epoch: 7550, cost 14.21 s 2023-09-05 10:11:40,026 44k INFO ====> Epoch: 7551, cost 14.51 s 2023-09-05 10:11:54,825 44k INFO ====> Epoch: 7552, cost 14.80 s 2023-09-05 10:12:00,578 44k INFO Train Epoch: 7553 [33%] 2023-09-05 10:12:00,579 44k INFO Losses: [1.9803125858306885, 3.1207408905029297, 9.16118049621582, 14.446868896484375, 0.5746681094169617], step: 158600, lr: 3.89045488603042e-05, reference_loss: 29.283771514892578 2023-09-05 10:12:09,865 44k INFO ====> Epoch: 7553, cost 15.04 s 2023-09-05 10:12:24,384 44k INFO ====> Epoch: 7554, cost 14.52 s 2023-09-05 10:12:38,848 44k INFO ====> Epoch: 7555, cost 14.46 s 2023-09-05 10:12:53,466 44k INFO ====> Epoch: 7556, cost 14.62 s 2023-09-05 10:13:08,165 44k INFO ====> Epoch: 7557, cost 14.70 s 2023-09-05 10:13:22,630 44k INFO ====> Epoch: 7558, cost 14.46 s 2023-09-05 10:13:37,042 44k INFO ====> Epoch: 7559, cost 14.41 s 2023-09-05 10:13:51,674 44k INFO ====> Epoch: 7560, cost 14.63 s 2023-09-05 10:14:06,376 44k INFO ====> Epoch: 7561, cost 14.70 s 2023-09-05 10:14:19,619 44k INFO Train Epoch: 7562 [86%] 2023-09-05 10:14:19,619 44k INFO Losses: [2.101958990097046, 2.782409191131592, 9.801591873168945, 14.723531723022461, 0.5120488405227661], step: 158800, lr: 3.886080312026351e-05, reference_loss: 29.921541213989258 2023-09-05 10:14:20,949 44k INFO ====> Epoch: 7562, cost 14.57 s 2023-09-05 10:14:35,343 44k INFO ====> Epoch: 7563, cost 14.39 s 2023-09-05 10:14:49,968 44k INFO ====> Epoch: 7564, cost 14.62 s 2023-09-05 10:15:04,483 44k INFO ====> Epoch: 7565, cost 14.52 s 2023-09-05 10:15:19,050 44k INFO ====> Epoch: 7566, cost 14.57 s 2023-09-05 10:15:33,477 44k INFO ====> Epoch: 7567, cost 14.43 s 2023-09-05 10:15:48,077 44k INFO ====> Epoch: 7568, cost 14.60 s 2023-09-05 10:16:02,704 44k INFO ====> Epoch: 7569, cost 14.63 s 2023-09-05 10:16:17,287 44k INFO ====> Epoch: 7570, cost 14.58 s 2023-09-05 10:16:31,727 44k INFO ====> Epoch: 7571, cost 14.44 s 2023-09-05 10:16:38,053 44k INFO Train Epoch: 7572 [38%] 2023-09-05 10:16:38,054 44k INFO Losses: [2.1953349113464355, 2.6532702445983887, 7.931266784667969, 14.899303436279297, 0.3959393799304962], step: 159000, lr: 3.881225443125935e-05, reference_loss: 28.075115203857422 2023-09-05 10:16:46,516 44k INFO ====> Epoch: 7572, cost 14.79 s 2023-09-05 10:17:01,142 44k INFO ====> Epoch: 7573, cost 14.63 s 2023-09-05 10:17:15,791 44k INFO ====> Epoch: 7574, cost 14.65 s 2023-09-05 10:17:30,159 44k INFO ====> Epoch: 7575, cost 14.37 s 2023-09-05 10:17:44,569 44k INFO ====> Epoch: 7576, cost 14.41 s 2023-09-05 10:17:59,408 44k INFO ====> Epoch: 7577, cost 14.84 s 2023-09-05 10:18:13,992 44k INFO ====> Epoch: 7578, cost 14.58 s 2023-09-05 10:18:28,367 44k INFO ====> Epoch: 7579, cost 14.38 s 2023-09-05 10:18:42,803 44k INFO ====> Epoch: 7580, cost 14.44 s 2023-09-05 10:18:56,825 44k INFO Train Epoch: 7581 [90%] 2023-09-05 10:18:56,826 44k INFO Losses: [2.211369037628174, 2.5441391468048096, 9.28738021850586, 15.49646282196045, 0.6910243630409241], step: 159200, lr: 3.876861247055085e-05, reference_loss: 30.230377197265625 2023-09-05 10:19:02,295 44k INFO Saving model and optimizer state at iteration 7581 to ./logs\44k\G_159200.pth 2023-09-05 10:19:02,958 44k INFO Saving model and optimizer state at iteration 7581 to ./logs\44k\D_159200.pth 2023-09-05 10:19:04,562 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_156800.pth 2023-09-05 10:19:04,596 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_156800.pth 2023-09-05 10:19:05,044 44k INFO ====> Epoch: 7581, cost 22.24 s 2023-09-05 10:19:19,295 44k INFO ====> Epoch: 7582, cost 14.25 s 2023-09-05 10:19:33,749 44k INFO ====> Epoch: 7583, cost 14.45 s 2023-09-05 10:19:48,233 44k INFO ====> Epoch: 7584, cost 14.48 s 2023-09-05 10:20:02,935 44k INFO ====> Epoch: 7585, cost 14.70 s 2023-09-05 10:20:17,559 44k INFO ====> Epoch: 7586, cost 14.62 s 2023-09-05 10:20:31,946 44k INFO ====> Epoch: 7587, cost 14.39 s 2023-09-05 10:20:46,560 44k INFO ====> Epoch: 7588, cost 14.61 s 2023-09-05 10:21:01,174 44k INFO ====> Epoch: 7589, cost 14.61 s 2023-09-05 10:21:15,701 44k INFO ====> Epoch: 7590, cost 14.53 s 2023-09-05 10:21:22,708 44k INFO Train Epoch: 7591 [43%] 2023-09-05 10:21:22,708 44k INFO Losses: [1.8326029777526855, 3.281076669692993, 9.936391830444336, 15.431278228759766, 0.533044695854187], step: 159400, lr: 3.872017895505888e-05, reference_loss: 31.014394760131836 2023-09-05 10:21:30,329 44k INFO ====> Epoch: 7591, cost 14.63 s 2023-09-05 10:21:44,719 44k INFO ====> Epoch: 7592, cost 14.39 s 2023-09-05 10:21:59,309 44k INFO ====> Epoch: 7593, cost 14.59 s 2023-09-05 10:22:13,799 44k INFO ====> Epoch: 7594, cost 14.49 s 2023-09-05 10:22:28,238 44k INFO ====> Epoch: 7595, cost 14.44 s 2023-09-05 10:22:42,636 44k INFO ====> Epoch: 7596, cost 14.40 s 2023-09-05 10:22:57,321 44k INFO ====> Epoch: 7597, cost 14.69 s 2023-09-05 10:23:12,101 44k INFO ====> Epoch: 7598, cost 14.78 s 2023-09-05 10:23:26,464 44k INFO ====> Epoch: 7599, cost 14.36 s 2023-09-05 10:23:41,061 44k INFO Train Epoch: 7600 [95%] 2023-09-05 10:23:41,062 44k INFO Losses: [2.1138453483581543, 2.6785593032836914, 9.90666389465332, 14.999034881591797, 0.20468083024024963], step: 159600, lr: 3.8676640527483755e-05, reference_loss: 29.90278434753418 2023-09-05 10:23:41,321 44k INFO ====> Epoch: 7600, cost 14.86 s 2023-09-05 10:23:56,110 44k INFO ====> Epoch: 7601, cost 14.79 s 2023-09-05 10:24:10,898 44k INFO ====> Epoch: 7602, cost 14.79 s 2023-09-05 10:24:25,362 44k INFO ====> Epoch: 7603, cost 14.46 s 2023-09-05 10:24:39,793 44k INFO ====> Epoch: 7604, cost 14.43 s 2023-09-05 10:24:54,328 44k INFO ====> Epoch: 7605, cost 14.54 s 2023-09-05 10:25:08,844 44k INFO ====> Epoch: 7606, cost 14.52 s 2023-09-05 10:25:23,376 44k INFO ====> Epoch: 7607, cost 14.53 s 2023-09-05 10:25:37,689 44k INFO ====> Epoch: 7608, cost 14.31 s 2023-09-05 10:25:52,386 44k INFO ====> Epoch: 7609, cost 14.70 s 2023-09-05 10:26:00,209 44k INFO Train Epoch: 7610 [48%] 2023-09-05 10:26:00,209 44k INFO Losses: [2.1252224445343018, 2.8915138244628906, 10.566340446472168, 15.66750431060791, 0.5397283434867859], step: 159800, lr: 3.86283219122744e-05, reference_loss: 31.790307998657227 2023-09-05 10:26:07,283 44k INFO ====> Epoch: 7610, cost 14.90 s 2023-09-05 10:26:21,877 44k INFO ====> Epoch: 7611, cost 14.59 s 2023-09-05 10:26:36,447 44k INFO ====> Epoch: 7612, cost 14.57 s 2023-09-05 10:26:50,914 44k INFO ====> Epoch: 7613, cost 14.47 s 2023-09-05 10:27:05,466 44k INFO ====> Epoch: 7614, cost 14.55 s 2023-09-05 10:27:19,796 44k INFO ====> Epoch: 7615, cost 14.33 s 2023-09-05 10:27:34,116 44k INFO ====> Epoch: 7616, cost 14.32 s 2023-09-05 10:27:48,700 44k INFO ====> Epoch: 7617, cost 14.58 s 2023-09-05 10:28:03,338 44k INFO ====> Epoch: 7618, cost 14.64 s 2023-09-05 10:28:17,867 44k INFO ====> Epoch: 7619, cost 14.53 s 2023-09-05 10:28:18,572 44k INFO Train Epoch: 7620 [0%] 2023-09-05 10:28:18,573 44k INFO Losses: [2.166896104812622, 2.9481849670410156, 9.251925468444824, 14.350210189819336, 0.574543833732605], step: 160000, lr: 3.858006366137135e-05, reference_loss: 29.291759490966797 2023-09-05 10:28:24,047 44k INFO Saving model and optimizer state at iteration 7620 to ./logs\44k\G_160000.pth 2023-09-05 10:28:24,805 44k INFO Saving model and optimizer state at iteration 7620 to ./logs\44k\D_160000.pth 2023-09-05 10:28:25,916 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_157600.pth 2023-09-05 10:28:25,954 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_157600.pth 2023-09-05 10:28:39,749 44k INFO ====> Epoch: 7620, cost 21.88 s 2023-09-05 10:28:54,291 44k INFO ====> Epoch: 7621, cost 14.54 s 2023-09-05 10:29:08,908 44k INFO ====> Epoch: 7622, cost 14.62 s 2023-09-05 10:29:23,301 44k INFO ====> Epoch: 7623, cost 14.39 s 2023-09-05 10:29:37,688 44k INFO ====> Epoch: 7624, cost 14.39 s 2023-09-05 10:29:52,385 44k INFO ====> Epoch: 7625, cost 14.70 s 2023-09-05 10:30:07,216 44k INFO ====> Epoch: 7626, cost 14.83 s 2023-09-05 10:30:21,807 44k INFO ====> Epoch: 7627, cost 14.59 s 2023-09-05 10:30:36,254 44k INFO ====> Epoch: 7628, cost 14.45 s 2023-09-05 10:30:44,735 44k INFO Train Epoch: 7629 [52%] 2023-09-05 10:30:44,735 44k INFO Losses: [2.1577463150024414, 3.0121617317199707, 8.457505226135254, 15.250641822814941, 0.6023722887039185], step: 160200, lr: 3.853668278470974e-05, reference_loss: 29.48042869567871 2023-09-05 10:30:51,110 44k INFO ====> Epoch: 7629, cost 14.86 s 2023-09-05 10:31:05,574 44k INFO ====> Epoch: 7630, cost 14.46 s 2023-09-05 10:31:20,036 44k INFO ====> Epoch: 7631, cost 14.46 s 2023-09-05 10:31:34,384 44k INFO ====> Epoch: 7632, cost 14.35 s 2023-09-05 10:31:48,878 44k INFO ====> Epoch: 7633, cost 14.49 s 2023-09-05 10:32:03,428 44k INFO ====> Epoch: 7634, cost 14.55 s 2023-09-05 10:32:17,819 44k INFO ====> Epoch: 7635, cost 14.39 s 2023-09-05 10:32:32,488 44k INFO ====> Epoch: 7636, cost 14.67 s 2023-09-05 10:32:47,014 44k INFO ====> Epoch: 7637, cost 14.53 s 2023-09-05 10:33:01,463 44k INFO ====> Epoch: 7638, cost 14.45 s 2023-09-05 10:33:02,894 44k INFO Train Epoch: 7639 [5%] 2023-09-05 10:33:02,894 44k INFO Losses: [2.0972166061401367, 2.8322155475616455, 12.312320709228516, 14.58798885345459, 0.5031175017356873], step: 160400, lr: 3.8488539018303856e-05, reference_loss: 32.33285903930664 2023-09-05 10:33:16,156 44k INFO ====> Epoch: 7639, cost 14.69 s 2023-09-05 10:33:30,596 44k INFO ====> Epoch: 7640, cost 14.44 s 2023-09-05 10:33:45,196 44k INFO ====> Epoch: 7641, cost 14.60 s 2023-09-05 10:33:59,824 44k INFO ====> Epoch: 7642, cost 14.63 s 2023-09-05 10:34:14,294 44k INFO ====> Epoch: 7643, cost 14.47 s 2023-09-05 10:34:28,658 44k INFO ====> Epoch: 7644, cost 14.36 s 2023-09-05 10:34:43,190 44k INFO ====> Epoch: 7645, cost 14.53 s 2023-09-05 10:34:57,802 44k INFO ====> Epoch: 7646, cost 14.61 s 2023-09-05 10:35:12,407 44k INFO ====> Epoch: 7647, cost 14.60 s 2023-09-05 10:35:21,338 44k INFO Train Epoch: 7648 [57%] 2023-09-05 10:35:21,338 44k INFO Losses: [1.8943581581115723, 3.2275960445404053, 11.582720756530762, 15.799596786499023, 0.4622747302055359], step: 160600, lr: 3.8445261055398106e-05, reference_loss: 32.966548919677734 2023-09-05 10:35:26,866 44k INFO ====> Epoch: 7648, cost 14.46 s 2023-09-05 10:35:41,403 44k INFO ====> Epoch: 7649, cost 14.54 s 2023-09-05 10:35:56,201 44k INFO ====> Epoch: 7650, cost 14.80 s 2023-09-05 10:36:11,137 44k INFO ====> Epoch: 7651, cost 14.94 s 2023-09-05 10:36:25,678 44k INFO ====> Epoch: 7652, cost 14.54 s 2023-09-05 10:36:40,041 44k INFO ====> Epoch: 7653, cost 14.36 s 2023-09-05 10:36:54,714 44k INFO ====> Epoch: 7654, cost 14.67 s 2023-09-05 10:37:09,273 44k INFO ====> Epoch: 7655, cost 14.56 s 2023-09-05 10:37:23,657 44k INFO ====> Epoch: 7656, cost 14.38 s 2023-09-05 10:37:37,976 44k INFO ====> Epoch: 7657, cost 14.32 s 2023-09-05 10:37:40,126 44k INFO Train Epoch: 7658 [10%] 2023-09-05 10:37:40,126 44k INFO Losses: [2.5417914390563965, 2.334057331085205, 9.347055435180664, 13.41207218170166, 0.710861325263977], step: 160800, lr: 3.839723150189439e-05, reference_loss: 28.345836639404297 2023-09-05 10:37:45,684 44k INFO Saving model and optimizer state at iteration 7658 to ./logs\44k\G_160800.pth 2023-09-05 10:37:46,312 44k INFO Saving model and optimizer state at iteration 7658 to ./logs\44k\D_160800.pth 2023-09-05 10:37:47,559 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_158400.pth 2023-09-05 10:37:47,595 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_158400.pth 2023-09-05 10:38:00,290 44k INFO ====> Epoch: 7658, cost 22.31 s 2023-09-05 10:38:14,878 44k INFO ====> Epoch: 7659, cost 14.59 s 2023-09-05 10:38:29,262 44k INFO ====> Epoch: 7660, cost 14.38 s 2023-09-05 10:38:43,756 44k INFO ====> Epoch: 7661, cost 14.49 s 2023-09-05 10:38:58,411 44k INFO ====> Epoch: 7662, cost 14.66 s 2023-09-05 10:39:12,971 44k INFO ====> Epoch: 7663, cost 14.56 s 2023-09-05 10:39:27,388 44k INFO ====> Epoch: 7664, cost 14.42 s 2023-09-05 10:39:41,692 44k INFO ====> Epoch: 7665, cost 14.30 s 2023-09-05 10:39:56,490 44k INFO ====> Epoch: 7666, cost 14.80 s 2023-09-05 10:40:06,457 44k INFO Train Epoch: 7667 [62%] 2023-09-05 10:40:06,457 44k INFO Losses: [2.086700201034546, 2.8065497875213623, 10.50430679321289, 15.232946395874023, 0.46302342414855957], step: 161000, lr: 3.835405620859909e-05, reference_loss: 31.09352684020996 2023-09-05 10:40:11,404 44k INFO ====> Epoch: 7667, cost 14.91 s 2023-09-05 10:40:25,783 44k INFO ====> Epoch: 7668, cost 14.38 s 2023-09-05 10:40:40,255 44k INFO ====> Epoch: 7669, cost 14.47 s 2023-09-05 10:40:54,869 44k INFO ====> Epoch: 7670, cost 14.61 s 2023-09-05 10:41:09,593 44k INFO ====> Epoch: 7671, cost 14.72 s 2023-09-05 10:41:24,126 44k INFO ====> Epoch: 7672, cost 14.53 s 2023-09-05 10:41:38,352 44k INFO ====> Epoch: 7673, cost 14.23 s 2023-09-05 10:41:53,089 44k INFO ====> Epoch: 7674, cost 14.74 s 2023-09-05 10:42:07,803 44k INFO ====> Epoch: 7675, cost 14.71 s 2023-09-05 10:42:22,282 44k INFO ====> Epoch: 7676, cost 14.48 s 2023-09-05 10:42:25,026 44k INFO Train Epoch: 7677 [14%] 2023-09-05 10:42:25,027 44k INFO Losses: [2.3687355518341064, 2.5057532787323, 9.525187492370605, 16.18218994140625, 0.5332611799240112], step: 161200, lr: 3.8306140597046824e-05, reference_loss: 31.115127563476562 2023-09-05 10:42:36,942 44k INFO ====> Epoch: 7677, cost 14.66 s 2023-09-05 10:42:51,488 44k INFO ====> Epoch: 7678, cost 14.55 s 2023-09-05 10:43:05,921 44k INFO ====> Epoch: 7679, cost 14.43 s 2023-09-05 10:43:20,309 44k INFO ====> Epoch: 7680, cost 14.39 s 2023-09-05 10:43:34,635 44k INFO ====> Epoch: 7681, cost 14.33 s 2023-09-05 10:43:49,108 44k INFO ====> Epoch: 7682, cost 14.47 s 2023-09-05 10:44:03,736 44k INFO ====> Epoch: 7683, cost 14.63 s 2023-09-05 10:44:18,208 44k INFO ====> Epoch: 7684, cost 14.47 s 2023-09-05 10:44:32,729 44k INFO ====> Epoch: 7685, cost 14.52 s 2023-09-05 10:44:43,124 44k INFO Train Epoch: 7686 [67%] 2023-09-05 10:44:43,125 44k INFO Losses: [2.3857014179229736, 2.377636671066284, 7.510050296783447, 13.562776565551758, 0.4845389723777771], step: 161400, lr: 3.8263067729795794e-05, reference_loss: 26.320703506469727 2023-09-05 10:44:47,460 44k INFO ====> Epoch: 7686, cost 14.73 s 2023-09-05 10:45:02,053 44k INFO ====> Epoch: 7687, cost 14.59 s 2023-09-05 10:45:16,573 44k INFO ====> Epoch: 7688, cost 14.52 s 2023-09-05 10:45:31,038 44k INFO ====> Epoch: 7689, cost 14.46 s 2023-09-05 10:45:45,402 44k INFO ====> Epoch: 7690, cost 14.36 s 2023-09-05 10:45:59,982 44k INFO ====> Epoch: 7691, cost 14.58 s 2023-09-05 10:46:14,650 44k INFO ====> Epoch: 7692, cost 14.67 s 2023-09-05 10:46:29,001 44k INFO ====> Epoch: 7693, cost 14.35 s 2023-09-05 10:46:43,623 44k INFO ====> Epoch: 7694, cost 14.62 s 2023-09-05 10:46:58,256 44k INFO ====> Epoch: 7695, cost 14.63 s 2023-09-05 10:47:01,836 44k INFO Train Epoch: 7696 [19%] 2023-09-05 10:47:01,836 44k INFO Losses: [2.3457250595092773, 2.4943599700927734, 8.681941986083984, 14.994384765625, 0.6574146747589111], step: 161600, lr: 3.821526578988708e-05, reference_loss: 29.173826217651367 2023-09-05 10:47:07,357 44k INFO Saving model and optimizer state at iteration 7696 to ./logs\44k\G_161600.pth 2023-09-05 10:47:08,070 44k INFO Saving model and optimizer state at iteration 7696 to ./logs\44k\D_161600.pth 2023-09-05 10:47:09,203 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_159200.pth 2023-09-05 10:47:09,240 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_159200.pth 2023-09-05 10:47:20,261 44k INFO ====> Epoch: 7696, cost 22.00 s 2023-09-05 10:47:34,721 44k INFO ====> Epoch: 7697, cost 14.46 s 2023-09-05 10:47:49,359 44k INFO ====> Epoch: 7698, cost 14.64 s 2023-09-05 10:48:04,184 44k INFO ====> Epoch: 7699, cost 14.83 s 2023-09-05 10:48:18,847 44k INFO ====> Epoch: 7700, cost 14.66 s 2023-09-05 10:48:33,308 44k INFO ====> Epoch: 7701, cost 14.46 s 2023-09-05 10:48:47,918 44k INFO ====> Epoch: 7702, cost 14.61 s 2023-09-05 10:49:02,273 44k INFO ====> Epoch: 7703, cost 14.36 s 2023-09-05 10:49:16,885 44k INFO ====> Epoch: 7704, cost 14.61 s 2023-09-05 10:49:28,066 44k INFO Train Epoch: 7705 [71%] 2023-09-05 10:49:28,067 44k INFO Losses: [2.218454599380493, 2.848283290863037, 7.978143215179443, 13.167282104492188, 0.7056747078895569], step: 161800, lr: 3.817229510569193e-05, reference_loss: 26.91783905029297 2023-09-05 10:49:31,549 44k INFO ====> Epoch: 7705, cost 14.66 s 2023-09-05 10:49:45,983 44k INFO ====> Epoch: 7706, cost 14.43 s 2023-09-05 10:50:00,659 44k INFO ====> Epoch: 7707, cost 14.68 s 2023-09-05 10:50:15,294 44k INFO ====> Epoch: 7708, cost 14.64 s 2023-09-05 10:50:29,817 44k INFO ====> Epoch: 7709, cost 14.52 s 2023-09-05 10:50:44,418 44k INFO ====> Epoch: 7710, cost 14.60 s 2023-09-05 10:50:59,053 44k INFO ====> Epoch: 7711, cost 14.64 s 2023-09-05 10:51:13,619 44k INFO ====> Epoch: 7712, cost 14.57 s 2023-09-05 10:51:27,990 44k INFO ====> Epoch: 7713, cost 14.37 s 2023-09-05 10:51:42,357 44k INFO ====> Epoch: 7714, cost 14.37 s 2023-09-05 10:51:46,586 44k INFO Train Epoch: 7715 [24%] 2023-09-05 10:51:46,586 44k INFO Losses: [2.2933316230773926, 2.6809170246124268, 10.684369087219238, 15.445720672607422, 0.46507367491722107], step: 162000, lr: 3.812460656776013e-05, reference_loss: 31.569414138793945 2023-09-05 10:51:57,058 44k INFO ====> Epoch: 7715, cost 14.70 s 2023-09-05 10:52:11,643 44k INFO ====> Epoch: 7716, cost 14.58 s 2023-09-05 10:52:26,187 44k INFO ====> Epoch: 7717, cost 14.54 s 2023-09-05 10:52:40,473 44k INFO ====> Epoch: 7718, cost 14.29 s 2023-09-05 10:52:55,114 44k INFO ====> Epoch: 7719, cost 14.64 s 2023-09-05 10:53:09,761 44k INFO ====> Epoch: 7720, cost 14.65 s 2023-09-05 10:53:24,051 44k INFO ====> Epoch: 7721, cost 14.29 s 2023-09-05 10:53:38,417 44k INFO ====> Epoch: 7722, cost 14.37 s 2023-09-05 10:53:53,170 44k INFO ====> Epoch: 7723, cost 14.75 s 2023-09-05 10:54:05,284 44k INFO Train Epoch: 7724 [76%] 2023-09-05 10:54:05,285 44k INFO Losses: [2.062471628189087, 2.726825475692749, 9.844047546386719, 13.904570579528809, 0.4565698802471161], step: 162200, lr: 3.8081737824208926e-05, reference_loss: 28.994483947753906 2023-09-05 10:54:08,232 44k INFO ====> Epoch: 7724, cost 15.06 s 2023-09-05 10:54:22,933 44k INFO ====> Epoch: 7725, cost 14.70 s 2023-09-05 10:54:37,413 44k INFO ====> Epoch: 7726, cost 14.48 s 2023-09-05 10:54:52,057 44k INFO ====> Epoch: 7727, cost 14.64 s 2023-09-05 10:55:06,783 44k INFO ====> Epoch: 7728, cost 14.73 s 2023-09-05 10:55:21,159 44k INFO ====> Epoch: 7729, cost 14.38 s 2023-09-05 10:55:35,500 44k INFO ====> Epoch: 7730, cost 14.34 s 2023-09-05 10:55:50,039 44k INFO ====> Epoch: 7731, cost 14.54 s 2023-09-05 10:56:04,737 44k INFO ====> Epoch: 7732, cost 14.70 s 2023-09-05 10:56:19,178 44k INFO ====> Epoch: 7733, cost 14.44 s 2023-09-05 10:56:24,137 44k INFO Train Epoch: 7734 [29%] 2023-09-05 10:56:24,137 44k INFO Losses: [2.280639410018921, 2.5897254943847656, 10.971043586730957, 16.74575424194336, 0.6900377869606018], step: 162400, lr: 3.80341624192271e-05, reference_loss: 33.277198791503906 2023-09-05 10:56:29,627 44k INFO Saving model and optimizer state at iteration 7734 to ./logs\44k\G_162400.pth 2023-09-05 10:56:30,301 44k INFO Saving model and optimizer state at iteration 7734 to ./logs\44k\D_162400.pth 2023-09-05 10:56:31,385 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_160000.pth 2023-09-05 10:56:31,425 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_160000.pth 2023-09-05 10:56:41,048 44k INFO ====> Epoch: 7734, cost 21.87 s 2023-09-05 10:56:55,734 44k INFO ====> Epoch: 7735, cost 14.69 s 2023-09-05 10:57:10,311 44k INFO ====> Epoch: 7736, cost 14.58 s 2023-09-05 10:57:24,767 44k INFO ====> Epoch: 7737, cost 14.46 s 2023-09-05 10:57:39,329 44k INFO ====> Epoch: 7738, cost 14.56 s 2023-09-05 10:57:54,382 44k INFO ====> Epoch: 7739, cost 15.05 s 2023-09-05 10:58:08,976 44k INFO ====> Epoch: 7740, cost 14.59 s 2023-09-05 10:58:23,553 44k INFO ====> Epoch: 7741, cost 14.58 s 2023-09-05 10:58:38,061 44k INFO ====> Epoch: 7742, cost 14.51 s 2023-09-05 10:58:50,853 44k INFO Train Epoch: 7743 [81%] 2023-09-05 10:58:50,854 44k INFO Losses: [2.133404016494751, 2.667086362838745, 9.731487274169922, 15.724541664123535, 0.6332011222839355], step: 162600, lr: 3.7991395374483e-05, reference_loss: 30.889720916748047 2023-09-05 10:58:52,921 44k INFO ====> Epoch: 7743, cost 14.86 s 2023-09-05 10:59:07,544 44k INFO ====> Epoch: 7744, cost 14.62 s 2023-09-05 10:59:21,985 44k INFO ====> Epoch: 7745, cost 14.44 s 2023-09-05 10:59:36,403 44k INFO ====> Epoch: 7746, cost 14.42 s 2023-09-05 10:59:51,139 44k INFO ====> Epoch: 7747, cost 14.74 s 2023-09-05 11:00:06,089 44k INFO ====> Epoch: 7748, cost 14.95 s 2023-09-05 11:00:20,497 44k INFO ====> Epoch: 7749, cost 14.41 s 2023-09-05 11:00:35,039 44k INFO ====> Epoch: 7750, cost 14.54 s 2023-09-05 11:00:49,642 44k INFO ====> Epoch: 7751, cost 14.60 s 2023-09-05 11:01:04,263 44k INFO ====> Epoch: 7752, cost 14.62 s 2023-09-05 11:01:09,891 44k INFO Train Epoch: 7753 [33%] 2023-09-05 11:01:09,891 44k INFO Losses: [2.2424263954162598, 2.9414234161376953, 11.120203971862793, 16.184141159057617, 0.5767253041267395], step: 162800, lr: 3.794393283406247e-05, reference_loss: 33.06492233276367 2023-09-05 11:01:19,041 44k INFO ====> Epoch: 7753, cost 14.78 s 2023-09-05 11:01:33,244 44k INFO ====> Epoch: 7754, cost 14.20 s 2023-09-05 11:01:47,645 44k INFO ====> Epoch: 7755, cost 14.40 s 2023-09-05 11:02:02,409 44k INFO ====> Epoch: 7756, cost 14.76 s 2023-09-05 11:02:16,924 44k INFO ====> Epoch: 7757, cost 14.52 s 2023-09-05 11:02:31,328 44k INFO ====> Epoch: 7758, cost 14.40 s 2023-09-05 11:02:45,945 44k INFO ====> Epoch: 7759, cost 14.62 s 2023-09-05 11:03:00,385 44k INFO ====> Epoch: 7760, cost 14.44 s 2023-09-05 11:03:14,880 44k INFO ====> Epoch: 7761, cost 14.49 s 2023-09-05 11:03:28,202 44k INFO Train Epoch: 7762 [86%] 2023-09-05 11:03:28,202 44k INFO Losses: [2.084859848022461, 2.7932288646698, 10.402033805847168, 14.506202697753906, 0.6598866581916809], step: 163000, lr: 3.790126724686235e-05, reference_loss: 30.446212768554688 2023-09-05 11:03:29,583 44k INFO ====> Epoch: 7762, cost 14.70 s 2023-09-05 11:03:43,780 44k INFO ====> Epoch: 7763, cost 14.20 s 2023-09-05 11:03:58,340 44k INFO ====> Epoch: 7764, cost 14.56 s 2023-09-05 11:04:13,104 44k INFO ====> Epoch: 7765, cost 14.76 s 2023-09-05 11:04:27,632 44k INFO ====> Epoch: 7766, cost 14.53 s 2023-09-05 11:04:42,055 44k INFO ====> Epoch: 7767, cost 14.42 s 2023-09-05 11:04:56,666 44k INFO ====> Epoch: 7768, cost 14.61 s 2023-09-05 11:05:11,329 44k INFO ====> Epoch: 7769, cost 14.66 s 2023-09-05 11:05:25,662 44k INFO ====> Epoch: 7770, cost 14.33 s 2023-09-05 11:05:39,982 44k INFO ====> Epoch: 7771, cost 14.32 s 2023-09-05 11:05:46,441 44k INFO Train Epoch: 7772 [38%] 2023-09-05 11:05:46,442 44k INFO Losses: [1.9512473344802856, 2.957024097442627, 9.08643627166748, 13.66070556640625, 0.5746944546699524], step: 163200, lr: 3.785391730325113e-05, reference_loss: 28.230106353759766 2023-09-05 11:05:51,924 44k INFO Saving model and optimizer state at iteration 7772 to ./logs\44k\G_163200.pth 2023-09-05 11:05:52,555 44k INFO Saving model and optimizer state at iteration 7772 to ./logs\44k\D_163200.pth 2023-09-05 11:05:53,701 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_160800.pth 2023-09-05 11:05:53,738 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_160800.pth 2023-09-05 11:06:01,967 44k INFO ====> Epoch: 7772, cost 21.99 s 2023-09-05 11:06:16,595 44k INFO ====> Epoch: 7773, cost 14.63 s 2023-09-05 11:06:31,025 44k INFO ====> Epoch: 7774, cost 14.43 s 2023-09-05 11:06:45,532 44k INFO ====> Epoch: 7775, cost 14.51 s 2023-09-05 11:07:00,243 44k INFO ====> Epoch: 7776, cost 14.71 s 2023-09-05 11:07:14,802 44k INFO ====> Epoch: 7777, cost 14.56 s 2023-09-05 11:07:29,061 44k INFO ====> Epoch: 7778, cost 14.26 s 2023-09-05 11:07:43,507 44k INFO ====> Epoch: 7779, cost 14.45 s 2023-09-05 11:07:58,017 44k INFO ====> Epoch: 7780, cost 14.51 s 2023-09-05 11:08:12,367 44k INFO Train Epoch: 7781 [90%] 2023-09-05 11:08:12,367 44k INFO Losses: [2.1994504928588867, 2.825068235397339, 9.774903297424316, 16.306503295898438, 0.8354392647743225], step: 163400, lr: 3.7811352932904196e-05, reference_loss: 31.94136619567871 2023-09-05 11:08:13,129 44k INFO ====> Epoch: 7781, cost 15.11 s 2023-09-05 11:08:27,710 44k INFO ====> Epoch: 7782, cost 14.58 s 2023-09-05 11:08:42,179 44k INFO ====> Epoch: 7783, cost 14.47 s 2023-09-05 11:08:56,770 44k INFO ====> Epoch: 7784, cost 14.59 s 2023-09-05 11:09:11,350 44k INFO ====> Epoch: 7785, cost 14.58 s 2023-09-05 11:09:25,715 44k INFO ====> Epoch: 7786, cost 14.37 s 2023-09-05 11:09:40,257 44k INFO ====> Epoch: 7787, cost 14.54 s 2023-09-05 11:09:54,703 44k INFO ====> Epoch: 7788, cost 14.45 s 2023-09-05 11:10:09,398 44k INFO ====> Epoch: 7789, cost 14.70 s 2023-09-05 11:10:23,784 44k INFO ====> Epoch: 7790, cost 14.39 s 2023-09-05 11:10:30,802 44k INFO Train Epoch: 7791 [43%] 2023-09-05 11:10:30,803 44k INFO Losses: [2.412144184112549, 2.45558500289917, 9.0806303024292, 15.814245223999023, 0.5388422012329102], step: 163600, lr: 3.776411531898548e-05, reference_loss: 30.30144500732422 2023-09-05 11:10:38,640 44k INFO ====> Epoch: 7791, cost 14.86 s 2023-09-05 11:10:53,272 44k INFO ====> Epoch: 7792, cost 14.63 s 2023-09-05 11:11:08,072 44k INFO ====> Epoch: 7793, cost 14.80 s 2023-09-05 11:11:22,314 44k INFO ====> Epoch: 7794, cost 14.24 s 2023-09-05 11:11:36,740 44k INFO ====> Epoch: 7795, cost 14.43 s 2023-09-05 11:11:51,342 44k INFO ====> Epoch: 7796, cost 14.60 s 2023-09-05 11:12:06,260 44k INFO ====> Epoch: 7797, cost 14.92 s 2023-09-05 11:12:20,927 44k INFO ====> Epoch: 7798, cost 14.67 s 2023-09-05 11:12:35,482 44k INFO ====> Epoch: 7799, cost 14.56 s 2023-09-05 11:12:49,880 44k INFO Train Epoch: 7800 [95%] 2023-09-05 11:12:49,881 44k INFO Losses: [2.0307059288024902, 3.0384767055511475, 10.752168655395508, 14.426656723022461, 0.49231022596359253], step: 163800, lr: 3.7721651925371956e-05, reference_loss: 30.740318298339844 2023-09-05 11:12:50,141 44k INFO ====> Epoch: 7800, cost 14.66 s 2023-09-05 11:13:04,732 44k INFO ====> Epoch: 7801, cost 14.59 s 2023-09-05 11:13:19,191 44k INFO ====> Epoch: 7802, cost 14.46 s 2023-09-05 11:13:33,531 44k INFO ====> Epoch: 7803, cost 14.34 s 2023-09-05 11:13:47,848 44k INFO ====> Epoch: 7804, cost 14.32 s 2023-09-05 11:14:02,376 44k INFO ====> Epoch: 7805, cost 14.53 s 2023-09-05 11:14:16,847 44k INFO ====> Epoch: 7806, cost 14.47 s 2023-09-05 11:14:31,189 44k INFO ====> Epoch: 7807, cost 14.34 s 2023-09-05 11:14:45,539 44k INFO ====> Epoch: 7808, cost 14.35 s 2023-09-05 11:15:00,023 44k INFO ====> Epoch: 7809, cost 14.48 s 2023-09-05 11:15:07,856 44k INFO Train Epoch: 7810 [48%] 2023-09-05 11:15:07,857 44k INFO Losses: [2.226879358291626, 2.6593682765960693, 11.823893547058105, 16.814929962158203, 0.6894153356552124], step: 164000, lr: 3.7674526374662653e-05, reference_loss: 34.21448516845703 2023-09-05 11:15:13,398 44k INFO Saving model and optimizer state at iteration 7810 to ./logs\44k\G_164000.pth 2023-09-05 11:15:14,072 44k INFO Saving model and optimizer state at iteration 7810 to ./logs\44k\D_164000.pth 2023-09-05 11:15:15,195 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_161600.pth 2023-09-05 11:15:15,228 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_161600.pth 2023-09-05 11:15:21,981 44k INFO ====> Epoch: 7810, cost 21.96 s 2023-09-05 11:15:36,162 44k INFO ====> Epoch: 7811, cost 14.18 s 2023-09-05 11:15:50,694 44k INFO ====> Epoch: 7812, cost 14.53 s 2023-09-05 11:16:05,092 44k INFO ====> Epoch: 7813, cost 14.40 s 2023-09-05 11:16:19,585 44k INFO ====> Epoch: 7814, cost 14.49 s 2023-09-05 11:16:34,106 44k INFO ====> Epoch: 7815, cost 14.52 s 2023-09-05 11:16:48,797 44k INFO ====> Epoch: 7816, cost 14.69 s 2023-09-05 11:17:03,469 44k INFO ====> Epoch: 7817, cost 14.67 s 2023-09-05 11:17:18,053 44k INFO ====> Epoch: 7818, cost 14.58 s 2023-09-05 11:17:32,292 44k INFO ====> Epoch: 7819, cost 14.24 s 2023-09-05 11:17:33,036 44k INFO Train Epoch: 7820 [0%] 2023-09-05 11:17:33,037 44k INFO Losses: [1.8536882400512695, 3.6863815784454346, 10.232414245605469, 14.743480682373047, 0.5514436364173889], step: 164200, lr: 3.7627459697767635e-05, reference_loss: 31.067407608032227 2023-09-05 11:17:47,149 44k INFO ====> Epoch: 7820, cost 14.86 s 2023-09-05 11:18:01,997 44k INFO ====> Epoch: 7821, cost 14.85 s 2023-09-05 11:18:16,624 44k INFO ====> Epoch: 7822, cost 14.63 s 2023-09-05 11:18:31,034 44k INFO ====> Epoch: 7823, cost 14.41 s 2023-09-05 11:18:45,682 44k INFO ====> Epoch: 7824, cost 14.65 s 2023-09-05 11:19:00,183 44k INFO ====> Epoch: 7825, cost 14.50 s 2023-09-05 11:19:14,743 44k INFO ====> Epoch: 7826, cost 14.56 s 2023-09-05 11:19:29,117 44k INFO ====> Epoch: 7827, cost 14.37 s 2023-09-05 11:19:43,545 44k INFO ====> Epoch: 7828, cost 14.43 s 2023-09-05 11:19:52,114 44k INFO Train Epoch: 7829 [52%] 2023-09-05 11:19:52,115 44k INFO Losses: [2.1909022331237793, 2.8795676231384277, 9.845519065856934, 16.15007209777832, 0.705618679523468], step: 164400, lr: 3.758514996488162e-05, reference_loss: 31.771678924560547 2023-09-05 11:19:58,528 44k INFO ====> Epoch: 7829, cost 14.98 s 2023-09-05 11:20:13,238 44k INFO ====> Epoch: 7830, cost 14.71 s 2023-09-05 11:20:27,550 44k INFO ====> Epoch: 7831, cost 14.31 s 2023-09-05 11:20:42,082 44k INFO ====> Epoch: 7832, cost 14.53 s 2023-09-05 11:20:56,646 44k INFO ====> Epoch: 7833, cost 14.56 s 2023-09-05 11:21:11,284 44k INFO ====> Epoch: 7834, cost 14.64 s 2023-09-05 11:21:25,770 44k INFO ====> Epoch: 7835, cost 14.49 s 2023-09-05 11:21:40,199 44k INFO ====> Epoch: 7836, cost 14.43 s 2023-09-05 11:21:54,785 44k INFO ====> Epoch: 7837, cost 14.59 s 2023-09-05 11:22:09,348 44k INFO ====> Epoch: 7838, cost 14.56 s 2023-09-05 11:22:10,754 44k INFO Train Epoch: 7839 [5%] 2023-09-05 11:22:10,755 44k INFO Losses: [2.1870131492614746, 2.733943223953247, 10.414198875427246, 15.737828254699707, 0.5445060729980469], step: 164600, lr: 3.753819494567698e-05, reference_loss: 31.617488861083984 2023-09-05 11:22:24,101 44k INFO ====> Epoch: 7839, cost 14.75 s 2023-09-05 11:22:38,565 44k INFO ====> Epoch: 7840, cost 14.46 s 2023-09-05 11:22:53,173 44k INFO ====> Epoch: 7841, cost 14.61 s 2023-09-05 11:23:07,805 44k INFO ====> Epoch: 7842, cost 14.63 s 2023-09-05 11:23:22,333 44k INFO ====> Epoch: 7843, cost 14.53 s 2023-09-05 11:23:36,720 44k INFO ====> Epoch: 7844, cost 14.39 s 2023-09-05 11:23:51,441 44k INFO ====> Epoch: 7845, cost 14.72 s 2023-09-05 11:24:06,230 44k INFO ====> Epoch: 7846, cost 14.79 s 2023-09-05 11:24:21,002 44k INFO ====> Epoch: 7847, cost 14.77 s 2023-09-05 11:24:30,241 44k INFO Train Epoch: 7848 [57%] 2023-09-05 11:24:30,241 44k INFO Losses: [2.2318007946014404, 2.7546374797821045, 9.024173736572266, 14.418843269348145, 0.7851362228393555], step: 164800, lr: 3.749598558544028e-05, reference_loss: 29.21459197998047 2023-09-05 11:24:35,714 44k INFO Saving model and optimizer state at iteration 7848 to ./logs\44k\G_164800.pth 2023-09-05 11:24:36,535 44k INFO Saving model and optimizer state at iteration 7848 to ./logs\44k\D_164800.pth 2023-09-05 11:24:37,676 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_162400.pth 2023-09-05 11:24:37,712 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_162400.pth 2023-09-05 11:24:43,070 44k INFO ====> Epoch: 7848, cost 22.07 s 2023-09-05 11:24:57,716 44k INFO ====> Epoch: 7849, cost 14.65 s 2023-09-05 11:25:12,332 44k INFO ====> Epoch: 7850, cost 14.62 s 2023-09-05 11:25:26,671 44k INFO ====> Epoch: 7851, cost 14.34 s 2023-09-05 11:25:41,015 44k INFO ====> Epoch: 7852, cost 14.34 s 2023-09-05 11:25:55,662 44k INFO ====> Epoch: 7853, cost 14.65 s 2023-09-05 11:26:10,332 44k INFO ====> Epoch: 7854, cost 14.67 s 2023-09-05 11:26:24,919 44k INFO ====> Epoch: 7855, cost 14.59 s 2023-09-05 11:26:39,310 44k INFO ====> Epoch: 7856, cost 14.39 s 2023-09-05 11:26:53,695 44k INFO ====> Epoch: 7857, cost 14.38 s 2023-09-05 11:26:55,828 44k INFO Train Epoch: 7858 [10%] 2023-09-05 11:26:55,828 44k INFO Losses: [2.324401378631592, 2.551758050918579, 9.608450889587402, 14.8390531539917, 0.6862005591392517], step: 165000, lr: 3.7449141959037136e-05, reference_loss: 30.009864807128906 2023-09-05 11:27:08,607 44k INFO ====> Epoch: 7858, cost 14.91 s 2023-09-05 11:27:22,903 44k INFO ====> Epoch: 7859, cost 14.30 s 2023-09-05 11:27:37,159 44k INFO ====> Epoch: 7860, cost 14.26 s 2023-09-05 11:27:51,629 44k INFO ====> Epoch: 7861, cost 14.47 s 2023-09-05 11:28:06,298 44k INFO ====> Epoch: 7862, cost 14.67 s 2023-09-05 11:28:20,699 44k INFO ====> Epoch: 7863, cost 14.40 s 2023-09-05 11:28:35,229 44k INFO ====> Epoch: 7864, cost 14.53 s 2023-09-05 11:28:49,816 44k INFO ====> Epoch: 7865, cost 14.59 s 2023-09-05 11:29:04,415 44k INFO ====> Epoch: 7866, cost 14.60 s 2023-09-05 11:29:14,250 44k INFO Train Epoch: 7867 [62%] 2023-09-05 11:29:14,251 44k INFO Losses: [2.1732187271118164, 2.594295024871826, 9.70106315612793, 16.224775314331055, 0.5434685945510864], step: 165200, lr: 3.740703273333271e-05, reference_loss: 31.236820220947266 2023-09-05 11:29:19,128 44k INFO ====> Epoch: 7867, cost 14.71 s 2023-09-05 11:29:33,569 44k INFO ====> Epoch: 7868, cost 14.44 s 2023-09-05 11:29:48,035 44k INFO ====> Epoch: 7869, cost 14.47 s 2023-09-05 11:30:02,627 44k INFO ====> Epoch: 7870, cost 14.59 s 2023-09-05 11:30:17,451 44k INFO ====> Epoch: 7871, cost 14.82 s 2023-09-05 11:30:31,962 44k INFO ====> Epoch: 7872, cost 14.51 s 2023-09-05 11:30:46,515 44k INFO ====> Epoch: 7873, cost 14.55 s 2023-09-05 11:31:01,075 44k INFO ====> Epoch: 7874, cost 14.56 s 2023-09-05 11:31:15,697 44k INFO ====> Epoch: 7875, cost 14.62 s 2023-09-05 11:31:30,141 44k INFO ====> Epoch: 7876, cost 14.44 s 2023-09-05 11:31:32,919 44k INFO Train Epoch: 7877 [14%] 2023-09-05 11:31:32,919 44k INFO Losses: [2.227184534072876, 2.819303274154663, 11.100552558898926, 15.332418441772461, 0.5762722492218018], step: 165400, lr: 3.736030023547056e-05, reference_loss: 32.05573272705078 2023-09-05 11:31:44,862 44k INFO ====> Epoch: 7877, cost 14.72 s 2023-09-05 11:31:59,453 44k INFO ====> Epoch: 7878, cost 14.59 s 2023-09-05 11:32:14,380 44k INFO ====> Epoch: 7879, cost 14.93 s 2023-09-05 11:32:28,965 44k INFO ====> Epoch: 7880, cost 14.59 s 2023-09-05 11:32:43,459 44k INFO ====> Epoch: 7881, cost 14.49 s 2023-09-05 11:32:58,108 44k INFO ====> Epoch: 7882, cost 14.65 s 2023-09-05 11:33:12,839 44k INFO ====> Epoch: 7883, cost 14.73 s 2023-09-05 11:33:27,275 44k INFO ====> Epoch: 7884, cost 14.44 s 2023-09-05 11:33:41,607 44k INFO ====> Epoch: 7885, cost 14.33 s 2023-09-05 11:33:52,177 44k INFO Train Epoch: 7886 [67%] 2023-09-05 11:33:52,179 44k INFO Losses: [2.291290760040283, 2.6181087493896484, 10.308432579040527, 16.155195236206055, 0.6168850660324097], step: 165600, lr: 3.731829090674625e-05, reference_loss: 31.989912033081055 2023-09-05 11:33:57,688 44k INFO Saving model and optimizer state at iteration 7886 to ./logs\44k\G_165600.pth 2023-09-05 11:33:58,348 44k INFO Saving model and optimizer state at iteration 7886 to ./logs\44k\D_165600.pth 2023-09-05 11:33:59,970 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_163200.pth 2023-09-05 11:34:00,007 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_163200.pth 2023-09-05 11:34:03,976 44k INFO ====> Epoch: 7886, cost 22.37 s 2023-09-05 11:34:18,422 44k INFO ====> Epoch: 7887, cost 14.45 s 2023-09-05 11:34:32,929 44k INFO ====> Epoch: 7888, cost 14.51 s 2023-09-05 11:34:47,527 44k INFO ====> Epoch: 7889, cost 14.60 s 2023-09-05 11:35:02,253 44k INFO ====> Epoch: 7890, cost 14.73 s 2023-09-05 11:35:16,744 44k INFO ====> Epoch: 7891, cost 14.49 s 2023-09-05 11:35:31,370 44k INFO ====> Epoch: 7892, cost 14.63 s 2023-09-05 11:35:45,745 44k INFO ====> Epoch: 7893, cost 14.38 s 2023-09-05 11:36:00,411 44k INFO ====> Epoch: 7894, cost 14.67 s 2023-09-05 11:36:15,058 44k INFO ====> Epoch: 7895, cost 14.65 s 2023-09-05 11:36:18,551 44k INFO Train Epoch: 7896 [19%] 2023-09-05 11:36:18,552 44k INFO Losses: [2.177903652191162, 2.7979159355163574, 8.703895568847656, 15.819829940795898, 0.6612682938575745], step: 165800, lr: 3.7271669273791534e-05, reference_loss: 30.16081428527832 2023-09-05 11:36:29,682 44k INFO ====> Epoch: 7896, cost 14.62 s 2023-09-05 11:36:44,133 44k INFO ====> Epoch: 7897, cost 14.45 s 2023-09-05 11:36:58,704 44k INFO ====> Epoch: 7898, cost 14.57 s 2023-09-05 11:37:13,220 44k INFO ====> Epoch: 7899, cost 14.52 s 2023-09-05 11:37:27,749 44k INFO ====> Epoch: 7900, cost 14.53 s 2023-09-05 11:37:42,319 44k INFO ====> Epoch: 7901, cost 14.57 s 2023-09-05 11:37:57,025 44k INFO ====> Epoch: 7902, cost 14.71 s 2023-09-05 11:38:11,559 44k INFO ====> Epoch: 7903, cost 14.53 s 2023-09-05 11:38:26,191 44k INFO ====> Epoch: 7904, cost 14.63 s 2023-09-05 11:38:37,468 44k INFO Train Epoch: 7905 [71%] 2023-09-05 11:38:37,469 44k INFO Losses: [2.1325645446777344, 2.763934373855591, 10.996505737304688, 14.329033851623535, 0.5833430290222168], step: 166000, lr: 3.7229759605058736e-05, reference_loss: 30.805381774902344 2023-09-05 11:38:40,950 44k INFO ====> Epoch: 7905, cost 14.76 s 2023-09-05 11:38:55,361 44k INFO ====> Epoch: 7906, cost 14.41 s 2023-09-05 11:39:09,879 44k INFO ====> Epoch: 7907, cost 14.52 s 2023-09-05 11:39:24,303 44k INFO ====> Epoch: 7908, cost 14.42 s 2023-09-05 11:39:38,654 44k INFO ====> Epoch: 7909, cost 14.35 s 2023-09-05 11:39:53,211 44k INFO ====> Epoch: 7910, cost 14.56 s 2023-09-05 11:40:07,620 44k INFO ====> Epoch: 7911, cost 14.41 s 2023-09-05 11:40:22,084 44k INFO ====> Epoch: 7912, cost 14.46 s 2023-09-05 11:40:36,555 44k INFO ====> Epoch: 7913, cost 14.47 s 2023-09-05 11:40:51,081 44k INFO ====> Epoch: 7914, cost 14.53 s 2023-09-05 11:40:55,314 44k INFO Train Epoch: 7915 [24%] 2023-09-05 11:40:55,314 44k INFO Losses: [2.2036383152008057, 2.7104434967041016, 7.354159355163574, 12.298474311828613, 0.5783243775367737], step: 166200, lr: 3.7183248574003306e-05, reference_loss: 25.14504051208496 2023-09-05 11:41:06,027 44k INFO ====> Epoch: 7915, cost 14.95 s 2023-09-05 11:41:20,472 44k INFO ====> Epoch: 7916, cost 14.45 s 2023-09-05 11:41:35,011 44k INFO ====> Epoch: 7917, cost 14.54 s 2023-09-05 11:41:49,545 44k INFO ====> Epoch: 7918, cost 14.53 s 2023-09-05 11:42:04,247 44k INFO ====> Epoch: 7919, cost 14.70 s 2023-09-05 11:42:18,691 44k INFO ====> Epoch: 7920, cost 14.44 s 2023-09-05 11:42:33,246 44k INFO ====> Epoch: 7921, cost 14.55 s 2023-09-05 11:42:47,796 44k INFO ====> Epoch: 7922, cost 14.55 s 2023-09-05 11:43:02,469 44k INFO ====> Epoch: 7923, cost 14.67 s 2023-09-05 11:43:14,562 44k INFO Train Epoch: 7924 [76%] 2023-09-05 11:43:14,562 44k INFO Losses: [1.961074709892273, 2.76815128326416, 11.817766189575195, 16.810195922851562, 0.6018518805503845], step: 166400, lr: 3.7141438328835626e-05, reference_loss: 33.959041595458984 2023-09-05 11:43:20,119 44k INFO Saving model and optimizer state at iteration 7924 to ./logs\44k\G_166400.pth 2023-09-05 11:43:20,846 44k INFO Saving model and optimizer state at iteration 7924 to ./logs\44k\D_166400.pth 2023-09-05 11:43:22,110 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_164000.pth 2023-09-05 11:43:22,145 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_164000.pth 2023-09-05 11:43:24,674 44k INFO ====> Epoch: 7924, cost 22.21 s 2023-09-05 11:43:38,975 44k INFO ====> Epoch: 7925, cost 14.30 s 2023-09-05 11:43:53,558 44k INFO ====> Epoch: 7926, cost 14.58 s 2023-09-05 11:44:08,252 44k INFO ====> Epoch: 7927, cost 14.69 s 2023-09-05 11:44:22,720 44k INFO ====> Epoch: 7928, cost 14.47 s 2023-09-05 11:44:37,383 44k INFO ====> Epoch: 7929, cost 14.66 s 2023-09-05 11:44:51,951 44k INFO ====> Epoch: 7930, cost 14.57 s 2023-09-05 11:45:06,569 44k INFO ====> Epoch: 7931, cost 14.62 s 2023-09-05 11:45:20,982 44k INFO ====> Epoch: 7932, cost 14.41 s 2023-09-05 11:45:35,502 44k INFO ====> Epoch: 7933, cost 14.52 s 2023-09-05 11:45:40,428 44k INFO Train Epoch: 7934 [29%] 2023-09-05 11:45:40,428 44k INFO Losses: [2.3235907554626465, 2.5486018657684326, 7.578188896179199, 14.11868953704834, 0.5672093033790588], step: 166600, lr: 3.709503763729526e-05, reference_loss: 27.136280059814453 2023-09-05 11:45:50,299 44k INFO ====> Epoch: 7934, cost 14.80 s 2023-09-05 11:46:04,868 44k INFO ====> Epoch: 7935, cost 14.57 s 2023-09-05 11:46:19,223 44k INFO ====> Epoch: 7936, cost 14.36 s 2023-09-05 11:46:33,786 44k INFO ====> Epoch: 7937, cost 14.56 s 2023-09-05 11:46:48,329 44k INFO ====> Epoch: 7938, cost 14.54 s 2023-09-05 11:47:02,873 44k INFO ====> Epoch: 7939, cost 14.54 s 2023-09-05 11:47:17,267 44k INFO ====> Epoch: 7940, cost 14.39 s 2023-09-05 11:47:31,571 44k INFO ====> Epoch: 7941, cost 14.30 s 2023-09-05 11:47:46,261 44k INFO ====> Epoch: 7942, cost 14.69 s 2023-09-05 11:47:59,278 44k INFO Train Epoch: 7943 [81%] 2023-09-05 11:47:59,278 44k INFO Losses: [1.9462387561798096, 3.216498851776123, 10.259531021118164, 15.232970237731934, 0.5819753408432007], step: 166800, lr: 3.705332657982719e-05, reference_loss: 31.237215042114258 2023-09-05 11:48:01,387 44k INFO ====> Epoch: 7943, cost 15.13 s 2023-09-05 11:48:16,085 44k INFO ====> Epoch: 7944, cost 14.70 s 2023-09-05 11:48:30,711 44k INFO ====> Epoch: 7945, cost 14.63 s 2023-09-05 11:48:45,185 44k INFO ====> Epoch: 7946, cost 14.47 s 2023-09-05 11:48:59,800 44k INFO ====> Epoch: 7947, cost 14.62 s 2023-09-05 11:49:14,430 44k INFO ====> Epoch: 7948, cost 14.63 s 2023-09-05 11:49:28,854 44k INFO ====> Epoch: 7949, cost 14.42 s 2023-09-05 11:49:43,224 44k INFO ====> Epoch: 7950, cost 14.37 s 2023-09-05 11:49:57,731 44k INFO ====> Epoch: 7951, cost 14.51 s 2023-09-05 11:50:12,246 44k INFO ====> Epoch: 7952, cost 14.51 s 2023-09-05 11:50:17,865 44k INFO Train Epoch: 7953 [33%] 2023-09-05 11:50:17,865 44k INFO Losses: [2.1344196796417236, 2.65985369682312, 9.414020538330078, 15.194214820861816, 0.5572256445884705], step: 167000, lr: 3.7007035966040164e-05, reference_loss: 29.959733963012695 2023-09-05 11:50:26,933 44k INFO ====> Epoch: 7953, cost 14.69 s 2023-09-05 11:50:41,394 44k INFO ====> Epoch: 7954, cost 14.46 s 2023-09-05 11:50:55,856 44k INFO ====> Epoch: 7955, cost 14.46 s 2023-09-05 11:51:10,431 44k INFO ====> Epoch: 7956, cost 14.57 s 2023-09-05 11:51:24,787 44k INFO ====> Epoch: 7957, cost 14.36 s 2023-09-05 11:51:39,183 44k INFO ====> Epoch: 7958, cost 14.40 s 2023-09-05 11:51:53,630 44k INFO ====> Epoch: 7959, cost 14.45 s 2023-09-05 11:52:08,090 44k INFO ====> Epoch: 7960, cost 14.46 s 2023-09-05 11:52:22,730 44k INFO ====> Epoch: 7961, cost 14.64 s 2023-09-05 11:52:36,024 44k INFO Train Epoch: 7962 [86%] 2023-09-05 11:52:36,024 44k INFO Losses: [1.8512839078903198, 3.0281310081481934, 11.804945945739746, 15.865004539489746, 0.6052203178405762], step: 167200, lr: 3.696542386096575e-05, reference_loss: 33.15458679199219 2023-09-05 11:52:41,540 44k INFO Saving model and optimizer state at iteration 7962 to ./logs\44k\G_167200.pth 2023-09-05 11:52:42,209 44k INFO Saving model and optimizer state at iteration 7962 to ./logs\44k\D_167200.pth 2023-09-05 11:52:43,420 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_164800.pth 2023-09-05 11:52:43,456 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_164800.pth 2023-09-05 11:52:44,603 44k INFO ====> Epoch: 7962, cost 21.87 s 2023-09-05 11:52:59,382 44k INFO ====> Epoch: 7963, cost 14.78 s 2023-09-05 11:53:13,872 44k INFO ====> Epoch: 7964, cost 14.49 s 2023-09-05 11:53:28,291 44k INFO ====> Epoch: 7965, cost 14.42 s 2023-09-05 11:53:42,688 44k INFO ====> Epoch: 7966, cost 14.40 s 2023-09-05 11:53:57,512 44k INFO ====> Epoch: 7967, cost 14.82 s 2023-09-05 11:54:12,193 44k INFO ====> Epoch: 7968, cost 14.68 s 2023-09-05 11:54:26,691 44k INFO ====> Epoch: 7969, cost 14.50 s 2023-09-05 11:54:41,297 44k INFO ====> Epoch: 7970, cost 14.61 s 2023-09-05 11:54:55,791 44k INFO ====> Epoch: 7971, cost 14.49 s 2023-09-05 11:55:02,189 44k INFO Train Epoch: 7972 [38%] 2023-09-05 11:55:02,189 44k INFO Losses: [2.128709554672241, 2.845737934112549, 11.6596040725708, 15.708751678466797, 0.5914753675460815], step: 167400, lr: 3.691924306379131e-05, reference_loss: 32.93428039550781 2023-09-05 11:55:10,786 44k INFO ====> Epoch: 7972, cost 15.00 s 2023-09-05 11:55:25,232 44k INFO ====> Epoch: 7973, cost 14.45 s 2023-09-05 11:55:39,642 44k INFO ====> Epoch: 7974, cost 14.41 s 2023-09-05 11:55:54,346 44k INFO ====> Epoch: 7975, cost 14.70 s 2023-09-05 11:56:09,057 44k INFO ====> Epoch: 7976, cost 14.71 s 2023-09-05 11:56:23,548 44k INFO ====> Epoch: 7977, cost 14.49 s 2023-09-05 11:56:38,020 44k INFO ====> Epoch: 7978, cost 14.47 s 2023-09-05 11:56:52,498 44k INFO ====> Epoch: 7979, cost 14.48 s 2023-09-05 11:57:07,145 44k INFO ====> Epoch: 7980, cost 14.65 s 2023-09-05 11:57:21,242 44k INFO Train Epoch: 7981 [90%] 2023-09-05 11:57:21,243 44k INFO Losses: [2.403050661087036, 2.487541675567627, 8.557637214660645, 15.478240966796875, 0.49630776047706604], step: 167600, lr: 3.687772967636283e-05, reference_loss: 29.422779083251953 2023-09-05 11:57:21,907 44k INFO ====> Epoch: 7981, cost 14.76 s 2023-09-05 11:57:36,138 44k INFO ====> Epoch: 7982, cost 14.23 s 2023-09-05 11:57:50,793 44k INFO ====> Epoch: 7983, cost 14.66 s 2023-09-05 11:58:05,302 44k INFO ====> Epoch: 7984, cost 14.51 s 2023-09-05 11:58:19,766 44k INFO ====> Epoch: 7985, cost 14.46 s 2023-09-05 11:58:34,193 44k INFO ====> Epoch: 7986, cost 14.43 s 2023-09-05 11:58:48,849 44k INFO ====> Epoch: 7987, cost 14.66 s 2023-09-05 11:59:03,483 44k INFO ====> Epoch: 7988, cost 14.63 s 2023-09-05 11:59:17,885 44k INFO ====> Epoch: 7989, cost 14.40 s 2023-09-05 11:59:32,270 44k INFO ====> Epoch: 7990, cost 14.39 s 2023-09-05 11:59:39,265 44k INFO Train Epoch: 7991 [43%] 2023-09-05 11:59:39,265 44k INFO Losses: [1.9190974235534668, 3.103510856628418, 10.18021297454834, 15.747604370117188, 0.5205870866775513], step: 167800, lr: 3.683165843527971e-05, reference_loss: 31.47101402282715 2023-09-05 11:59:47,143 44k INFO ====> Epoch: 7991, cost 14.87 s 2023-09-05 12:00:01,932 44k INFO ====> Epoch: 7992, cost 14.79 s 2023-09-05 12:00:16,552 44k INFO ====> Epoch: 7993, cost 14.62 s 2023-09-05 12:00:30,889 44k INFO ====> Epoch: 7994, cost 14.34 s 2023-09-05 12:00:45,323 44k INFO ====> Epoch: 7995, cost 14.43 s 2023-09-05 12:00:59,910 44k INFO ====> Epoch: 7996, cost 14.59 s 2023-09-05 12:01:14,321 44k INFO ====> Epoch: 7997, cost 14.41 s 2023-09-05 12:01:28,754 44k INFO ====> Epoch: 7998, cost 14.43 s 2023-09-05 12:01:43,086 44k INFO ====> Epoch: 7999, cost 14.33 s 2023-09-05 12:01:57,771 44k INFO Train Epoch: 8000 [95%] 2023-09-05 12:01:57,772 44k INFO Losses: [2.002589225769043, 2.723959445953369, 10.229707717895508, 15.328473091125488, 0.2721633017063141], step: 168000, lr: 3.679024353130632e-05, reference_loss: 30.55689239501953 2023-09-05 12:02:03,268 44k INFO Saving model and optimizer state at iteration 8000 to ./logs\44k\G_168000.pth 2023-09-05 12:02:03,831 44k INFO Saving model and optimizer state at iteration 8000 to ./logs\44k\D_168000.pth 2023-09-05 12:02:05,040 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_165600.pth 2023-09-05 12:02:05,077 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_165600.pth 2023-09-05 12:02:05,077 44k INFO ====> Epoch: 8000, cost 21.99 s 2023-09-05 12:02:19,725 44k INFO ====> Epoch: 8001, cost 14.65 s 2023-09-05 12:02:34,166 44k INFO ====> Epoch: 8002, cost 14.44 s 2023-09-05 12:02:48,781 44k INFO ====> Epoch: 8003, cost 14.62 s 2023-09-05 12:03:03,526 44k INFO ====> Epoch: 8004, cost 14.74 s 2023-09-05 12:03:18,026 44k INFO ====> Epoch: 8005, cost 14.50 s 2023-09-05 12:03:32,464 44k INFO ====> Epoch: 8006, cost 14.44 s 2023-09-05 12:03:46,898 44k INFO ====> Epoch: 8007, cost 14.43 s 2023-09-05 12:04:01,419 44k INFO ====> Epoch: 8008, cost 14.52 s 2023-09-05 12:04:15,959 44k INFO ====> Epoch: 8009, cost 14.54 s 2023-09-05 12:04:23,775 44k INFO Train Epoch: 8010 [48%] 2023-09-05 12:04:23,776 44k INFO Losses: [2.0272490978240967, 2.9055087566375732, 10.943410873413086, 16.419696807861328, 0.7289064526557922], step: 168200, lr: 3.6744281586411326e-05, reference_loss: 33.02477264404297 2023-09-05 12:04:30,624 44k INFO ====> Epoch: 8010, cost 14.67 s 2023-09-05 12:04:45,317 44k INFO ====> Epoch: 8011, cost 14.69 s 2023-09-05 12:05:00,090 44k INFO ====> Epoch: 8012, cost 14.77 s 2023-09-05 12:05:14,738 44k INFO ====> Epoch: 8013, cost 14.65 s 2023-09-05 12:05:29,101 44k INFO ====> Epoch: 8014, cost 14.36 s 2023-09-05 12:05:43,592 44k INFO ====> Epoch: 8015, cost 14.49 s 2023-09-05 12:05:58,552 44k INFO ====> Epoch: 8016, cost 14.96 s 2023-09-05 12:06:13,279 44k INFO ====> Epoch: 8017, cost 14.73 s 2023-09-05 12:06:27,728 44k INFO ====> Epoch: 8018, cost 14.45 s 2023-09-05 12:06:42,209 44k INFO ====> Epoch: 8019, cost 14.48 s 2023-09-05 12:06:42,954 44k INFO Train Epoch: 8020 [0%] 2023-09-05 12:06:42,954 44k INFO Losses: [2.223628520965576, 2.6609511375427246, 8.123169898986816, 13.649637222290039, 0.5719742178916931], step: 168400, lr: 3.6698377061641235e-05, reference_loss: 27.229360580444336 2023-09-05 12:06:57,005 44k INFO ====> Epoch: 8020, cost 14.80 s 2023-09-05 12:07:11,658 44k INFO ====> Epoch: 8021, cost 14.65 s 2023-09-05 12:07:26,030 44k INFO ====> Epoch: 8022, cost 14.37 s 2023-09-05 12:07:40,404 44k INFO ====> Epoch: 8023, cost 14.37 s 2023-09-05 12:07:55,238 44k INFO ====> Epoch: 8024, cost 14.83 s 2023-09-05 12:08:09,984 44k INFO ====> Epoch: 8025, cost 14.75 s 2023-09-05 12:08:24,334 44k INFO ====> Epoch: 8026, cost 14.35 s 2023-09-05 12:08:38,869 44k INFO ====> Epoch: 8027, cost 14.53 s 2023-09-05 12:08:53,506 44k INFO ====> Epoch: 8028, cost 14.64 s 2023-09-05 12:09:01,877 44k INFO Train Epoch: 8029 [52%] 2023-09-05 12:09:01,877 44k INFO Losses: [2.098979949951172, 2.9084737300872803, 9.25926685333252, 15.98310375213623, 0.464297890663147], step: 168600, lr: 3.665711202426427e-05, reference_loss: 30.714122772216797 2023-09-05 12:09:08,170 44k INFO ====> Epoch: 8029, cost 14.66 s 2023-09-05 12:09:22,525 44k INFO ====> Epoch: 8030, cost 14.36 s 2023-09-05 12:09:36,819 44k INFO ====> Epoch: 8031, cost 14.29 s 2023-09-05 12:09:51,246 44k INFO ====> Epoch: 8032, cost 14.43 s 2023-09-05 12:10:05,839 44k INFO ====> Epoch: 8033, cost 14.59 s 2023-09-05 12:10:20,311 44k INFO ====> Epoch: 8034, cost 14.47 s 2023-09-05 12:10:34,724 44k INFO ====> Epoch: 8035, cost 14.41 s 2023-09-05 12:10:49,284 44k INFO ====> Epoch: 8036, cost 14.56 s 2023-09-05 12:11:03,966 44k INFO ====> Epoch: 8037, cost 14.68 s 2023-09-05 12:11:18,470 44k INFO ====> Epoch: 8038, cost 14.50 s 2023-09-05 12:11:19,882 44k INFO Train Epoch: 8039 [5%] 2023-09-05 12:11:19,882 44k INFO Losses: [2.4210615158081055, 2.637932062149048, 9.044493675231934, 15.242406845092773, 0.5964869260787964], step: 168800, lr: 3.6611316400176186e-05, reference_loss: 29.942380905151367 2023-09-05 12:11:25,368 44k INFO Saving model and optimizer state at iteration 8039 to ./logs\44k\G_168800.pth 2023-09-05 12:11:26,301 44k INFO Saving model and optimizer state at iteration 8039 to ./logs\44k\D_168800.pth 2023-09-05 12:11:27,485 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_166400.pth 2023-09-05 12:11:27,526 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_166400.pth 2023-09-05 12:11:40,668 44k INFO ====> Epoch: 8039, cost 22.20 s 2023-09-05 12:11:55,380 44k INFO ====> Epoch: 8040, cost 14.71 s 2023-09-05 12:12:10,093 44k INFO ====> Epoch: 8041, cost 14.71 s 2023-09-05 12:12:24,708 44k INFO ====> Epoch: 8042, cost 14.61 s 2023-09-05 12:12:39,197 44k INFO ====> Epoch: 8043, cost 14.49 s 2023-09-05 12:12:53,700 44k INFO ====> Epoch: 8044, cost 14.50 s 2023-09-05 12:13:08,303 44k INFO ====> Epoch: 8045, cost 14.60 s 2023-09-05 12:13:22,727 44k INFO ====> Epoch: 8046, cost 14.42 s 2023-09-05 12:13:36,978 44k INFO ====> Epoch: 8047, cost 14.25 s 2023-09-05 12:13:46,050 44k INFO Train Epoch: 8048 [57%] 2023-09-05 12:13:46,050 44k INFO Losses: [2.2311348915100098, 2.589212417602539, 8.514745712280273, 14.528495788574219, 0.6008278131484985], step: 169000, lr: 3.657014925708603e-05, reference_loss: 28.46441650390625 2023-09-05 12:13:51,687 44k INFO ====> Epoch: 8048, cost 14.71 s 2023-09-05 12:14:06,486 44k INFO ====> Epoch: 8049, cost 14.80 s 2023-09-05 12:14:21,003 44k INFO ====> Epoch: 8050, cost 14.52 s 2023-09-05 12:14:35,490 44k INFO ====> Epoch: 8051, cost 14.49 s 2023-09-05 12:14:50,014 44k INFO ====> Epoch: 8052, cost 14.52 s 2023-09-05 12:15:04,692 44k INFO ====> Epoch: 8053, cost 14.68 s 2023-09-05 12:15:19,086 44k INFO ====> Epoch: 8054, cost 14.39 s 2023-09-05 12:15:33,336 44k INFO ====> Epoch: 8055, cost 14.25 s 2023-09-05 12:15:47,665 44k INFO ====> Epoch: 8056, cost 14.33 s 2023-09-05 12:16:02,307 44k INFO ====> Epoch: 8057, cost 14.64 s 2023-09-05 12:16:04,417 44k INFO Train Epoch: 8058 [10%] 2023-09-05 12:16:04,418 44k INFO Losses: [2.1831812858581543, 2.8699774742126465, 9.884467124938965, 15.350589752197266, 0.5510289669036865], step: 169200, lr: 3.652446227533159e-05, reference_loss: 30.839244842529297 2023-09-05 12:16:17,113 44k INFO ====> Epoch: 8058, cost 14.81 s 2023-09-05 12:16:31,570 44k INFO ====> Epoch: 8059, cost 14.46 s 2023-09-05 12:16:46,137 44k INFO ====> Epoch: 8060, cost 14.57 s 2023-09-05 12:17:00,722 44k INFO ====> Epoch: 8061, cost 14.58 s 2023-09-05 12:17:15,216 44k INFO ====> Epoch: 8062, cost 14.49 s 2023-09-05 12:17:29,529 44k INFO ====> Epoch: 8063, cost 14.31 s 2023-09-05 12:17:44,081 44k INFO ====> Epoch: 8064, cost 14.55 s 2023-09-05 12:17:59,039 44k INFO ====> Epoch: 8065, cost 14.96 s 2023-09-05 12:18:13,704 44k INFO ====> Epoch: 8066, cost 14.66 s 2023-09-05 12:18:23,502 44k INFO Train Epoch: 8067 [62%] 2023-09-05 12:18:23,503 44k INFO Losses: [2.4157984256744385, 2.4704558849334717, 9.337583541870117, 14.847892761230469, 0.5073062181472778], step: 169400, lr: 3.648339279429068e-05, reference_loss: 29.579038619995117 2023-09-05 12:18:28,430 44k INFO ====> Epoch: 8067, cost 14.73 s 2023-09-05 12:18:42,825 44k INFO ====> Epoch: 8068, cost 14.40 s 2023-09-05 12:18:57,359 44k INFO ====> Epoch: 8069, cost 14.53 s 2023-09-05 12:19:12,069 44k INFO ====> Epoch: 8070, cost 14.71 s 2023-09-05 12:19:26,407 44k INFO ====> Epoch: 8071, cost 14.34 s 2023-09-05 12:19:40,836 44k INFO ====> Epoch: 8072, cost 14.43 s 2023-09-05 12:19:55,604 44k INFO ====> Epoch: 8073, cost 14.77 s 2023-09-05 12:20:10,106 44k INFO ====> Epoch: 8074, cost 14.50 s 2023-09-05 12:20:24,556 44k INFO ====> Epoch: 8075, cost 14.45 s 2023-09-05 12:20:39,010 44k INFO ====> Epoch: 8076, cost 14.45 s 2023-09-05 12:20:41,791 44k INFO Train Epoch: 8077 [14%] 2023-09-05 12:20:41,792 44k INFO Losses: [2.1530752182006836, 2.890275478363037, 10.898307800292969, 14.63056468963623, 0.6385971903800964], step: 169600, lr: 3.6437814197134434e-05, reference_loss: 31.2108211517334 2023-09-05 12:20:47,288 44k INFO Saving model and optimizer state at iteration 8077 to ./logs\44k\G_169600.pth 2023-09-05 12:20:47,933 44k INFO Saving model and optimizer state at iteration 8077 to ./logs\44k\D_169600.pth 2023-09-05 12:20:49,166 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_167200.pth 2023-09-05 12:20:49,205 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_167200.pth 2023-09-05 12:21:00,945 44k INFO ====> Epoch: 8077, cost 21.93 s 2023-09-05 12:21:15,364 44k INFO ====> Epoch: 8078, cost 14.42 s 2023-09-05 12:21:29,768 44k INFO ====> Epoch: 8079, cost 14.40 s 2023-09-05 12:21:44,348 44k INFO ====> Epoch: 8080, cost 14.58 s 2023-09-05 12:21:59,062 44k INFO ====> Epoch: 8081, cost 14.71 s 2023-09-05 12:22:13,639 44k INFO ====> Epoch: 8082, cost 14.58 s 2023-09-05 12:22:27,999 44k INFO ====> Epoch: 8083, cost 14.36 s 2023-09-05 12:22:42,385 44k INFO ====> Epoch: 8084, cost 14.39 s 2023-09-05 12:22:56,861 44k INFO ====> Epoch: 8085, cost 14.48 s 2023-09-05 12:23:07,642 44k INFO Train Epoch: 8086 [67%] 2023-09-05 12:23:07,643 44k INFO Losses: [2.5319321155548096, 2.314546585083008, 8.36217212677002, 14.760891914367676, 0.5273934602737427], step: 169800, lr: 3.639684214645618e-05, reference_loss: 28.496936798095703 2023-09-05 12:23:11,804 44k INFO ====> Epoch: 8086, cost 14.94 s 2023-09-05 12:23:26,121 44k INFO ====> Epoch: 8087, cost 14.32 s 2023-09-05 12:23:40,441 44k INFO ====> Epoch: 8088, cost 14.32 s 2023-09-05 12:23:55,274 44k INFO ====> Epoch: 8089, cost 14.83 s 2023-09-05 12:24:10,077 44k INFO ====> Epoch: 8090, cost 14.80 s 2023-09-05 12:24:24,719 44k INFO ====> Epoch: 8091, cost 14.64 s 2023-09-05 12:24:39,154 44k INFO ====> Epoch: 8092, cost 14.43 s 2023-09-05 12:24:53,616 44k INFO ====> Epoch: 8093, cost 14.46 s 2023-09-05 12:25:08,247 44k INFO ====> Epoch: 8094, cost 14.63 s 2023-09-05 12:25:22,561 44k INFO ====> Epoch: 8095, cost 14.31 s 2023-09-05 12:25:26,038 44k INFO Train Epoch: 8096 [19%] 2023-09-05 12:25:26,039 44k INFO Losses: [1.9794533252716064, 3.0681629180908203, 10.751876831054688, 14.784577369689941, 0.5004616379737854], step: 170000, lr: 3.6351371676774086e-05, reference_loss: 31.084531784057617 2023-09-05 12:25:37,311 44k INFO ====> Epoch: 8096, cost 14.75 s 2023-09-05 12:25:51,952 44k INFO ====> Epoch: 8097, cost 14.64 s 2023-09-05 12:26:06,536 44k INFO ====> Epoch: 8098, cost 14.58 s 2023-09-05 12:26:20,859 44k INFO ====> Epoch: 8099, cost 14.32 s 2023-09-05 12:26:35,289 44k INFO ====> Epoch: 8100, cost 14.43 s 2023-09-05 12:26:49,813 44k INFO ====> Epoch: 8101, cost 14.52 s 2023-09-05 12:27:04,371 44k INFO ====> Epoch: 8102, cost 14.56 s 2023-09-05 12:27:18,917 44k INFO ====> Epoch: 8103, cost 14.55 s 2023-09-05 12:27:33,491 44k INFO ====> Epoch: 8104, cost 14.57 s 2023-09-05 12:27:44,804 44k INFO Train Epoch: 8105 [71%] 2023-09-05 12:27:44,804 44k INFO Losses: [2.130617380142212, 2.8035740852355957, 10.114636421203613, 14.471142768859863, 0.424482136964798], step: 170200, lr: 3.631049682532149e-05, reference_loss: 29.9444522857666 2023-09-05 12:27:48,319 44k INFO ====> Epoch: 8105, cost 14.83 s 2023-09-05 12:28:02,914 44k INFO ====> Epoch: 8106, cost 14.59 s 2023-09-05 12:28:17,451 44k INFO ====> Epoch: 8107, cost 14.54 s 2023-09-05 12:28:31,832 44k INFO ====> Epoch: 8108, cost 14.38 s 2023-09-05 12:28:46,424 44k INFO ====> Epoch: 8109, cost 14.59 s 2023-09-05 12:29:01,166 44k INFO ====> Epoch: 8110, cost 14.74 s 2023-09-05 12:29:15,957 44k INFO ====> Epoch: 8111, cost 14.79 s 2023-09-05 12:29:30,335 44k INFO ====> Epoch: 8112, cost 14.38 s 2023-09-05 12:29:44,890 44k INFO ====> Epoch: 8113, cost 14.56 s 2023-09-05 12:29:59,771 44k INFO ====> Epoch: 8114, cost 14.88 s 2023-09-05 12:30:04,067 44k INFO Train Epoch: 8115 [24%] 2023-09-05 12:30:04,067 44k INFO Losses: [2.126344680786133, 2.7890830039978027, 9.922355651855469, 14.6583833694458, 0.6923176050186157], step: 170400, lr: 3.626513422659949e-05, reference_loss: 30.18848419189453 2023-09-05 12:30:09,535 44k INFO Saving model and optimizer state at iteration 8115 to ./logs\44k\G_170400.pth 2023-09-05 12:30:10,232 44k INFO Saving model and optimizer state at iteration 8115 to ./logs\44k\D_170400.pth 2023-09-05 12:30:11,552 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_168000.pth 2023-09-05 12:30:11,589 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_168000.pth 2023-09-05 12:30:21,795 44k INFO ====> Epoch: 8115, cost 22.02 s 2023-09-05 12:30:36,236 44k INFO ====> Epoch: 8116, cost 14.44 s 2023-09-05 12:30:50,808 44k INFO ====> Epoch: 8117, cost 14.57 s 2023-09-05 12:31:05,416 44k INFO ====> Epoch: 8118, cost 14.61 s 2023-09-05 12:31:19,907 44k INFO ====> Epoch: 8119, cost 14.49 s 2023-09-05 12:31:34,310 44k INFO ====> Epoch: 8120, cost 14.40 s 2023-09-05 12:31:48,934 44k INFO ====> Epoch: 8121, cost 14.62 s 2023-09-05 12:32:03,664 44k INFO ====> Epoch: 8122, cost 14.73 s 2023-09-05 12:32:18,320 44k INFO ====> Epoch: 8123, cost 14.66 s 2023-09-05 12:32:30,317 44k INFO Train Epoch: 8124 [76%] 2023-09-05 12:32:30,317 44k INFO Losses: [2.087052583694458, 2.6416218280792236, 9.607731819152832, 16.392681121826172, 0.6983048319816589], step: 170600, lr: 3.6224356343783925e-05, reference_loss: 31.427392959594727 2023-09-05 12:32:33,095 44k INFO ====> Epoch: 8124, cost 14.78 s 2023-09-05 12:32:47,670 44k INFO ====> Epoch: 8125, cost 14.58 s 2023-09-05 12:33:02,202 44k INFO ====> Epoch: 8126, cost 14.53 s 2023-09-05 12:33:16,600 44k INFO ====> Epoch: 8127, cost 14.40 s 2023-09-05 12:33:30,985 44k INFO ====> Epoch: 8128, cost 14.39 s 2023-09-05 12:33:45,468 44k INFO ====> Epoch: 8129, cost 14.48 s 2023-09-05 12:34:00,001 44k INFO ====> Epoch: 8130, cost 14.53 s 2023-09-05 12:34:14,522 44k INFO ====> Epoch: 8131, cost 14.52 s 2023-09-05 12:34:28,969 44k INFO ====> Epoch: 8132, cost 14.45 s 2023-09-05 12:34:43,446 44k INFO ====> Epoch: 8133, cost 14.48 s 2023-09-05 12:34:48,410 44k INFO Train Epoch: 8134 [29%] 2023-09-05 12:34:48,410 44k INFO Losses: [1.9600856304168701, 2.91682505607605, 10.262864112854004, 14.98795223236084, 0.5145216584205627], step: 170800, lr: 3.6179101360116515e-05, reference_loss: 30.642250061035156 2023-09-05 12:34:58,203 44k INFO ====> Epoch: 8134, cost 14.76 s 2023-09-05 12:35:12,893 44k INFO ====> Epoch: 8135, cost 14.69 s 2023-09-05 12:35:27,432 44k INFO ====> Epoch: 8136, cost 14.54 s 2023-09-05 12:35:41,791 44k INFO ====> Epoch: 8137, cost 14.36 s 2023-09-05 12:35:56,540 44k INFO ====> Epoch: 8138, cost 14.75 s 2023-09-05 12:36:11,241 44k INFO ====> Epoch: 8139, cost 14.70 s 2023-09-05 12:36:25,738 44k INFO ====> Epoch: 8140, cost 14.50 s 2023-09-05 12:36:40,284 44k INFO ====> Epoch: 8141, cost 14.55 s 2023-09-05 12:36:54,883 44k INFO ====> Epoch: 8142, cost 14.60 s 2023-09-05 12:37:07,538 44k INFO Train Epoch: 8143 [81%] 2023-09-05 12:37:07,538 44k INFO Losses: [2.338926315307617, 2.379056215286255, 9.354653358459473, 14.7775297164917, 0.6791691780090332], step: 171000, lr: 3.6138420215896366e-05, reference_loss: 29.529335021972656 2023-09-05 12:37:09,639 44k INFO ====> Epoch: 8143, cost 14.76 s 2023-09-05 12:37:23,965 44k INFO ====> Epoch: 8144, cost 14.33 s 2023-09-05 12:37:38,420 44k INFO ====> Epoch: 8145, cost 14.45 s 2023-09-05 12:37:53,141 44k INFO ====> Epoch: 8146, cost 14.72 s 2023-09-05 12:38:07,776 44k INFO ====> Epoch: 8147, cost 14.63 s 2023-09-05 12:38:22,360 44k INFO ====> Epoch: 8148, cost 14.58 s 2023-09-05 12:38:36,846 44k INFO ====> Epoch: 8149, cost 14.49 s 2023-09-05 12:38:51,494 44k INFO ====> Epoch: 8150, cost 14.65 s 2023-09-05 12:39:05,988 44k INFO ====> Epoch: 8151, cost 14.49 s 2023-09-05 12:39:20,465 44k INFO ====> Epoch: 8152, cost 14.48 s 2023-09-05 12:39:26,014 44k INFO Train Epoch: 8153 [33%] 2023-09-05 12:39:26,015 44k INFO Losses: [2.0790107250213623, 2.867471694946289, 8.01615047454834, 14.110615730285645, 0.6416710615158081], step: 171200, lr: 3.609327259198511e-05, reference_loss: 27.714920043945312 2023-09-05 12:39:31,494 44k INFO Saving model and optimizer state at iteration 8153 to ./logs\44k\G_171200.pth 2023-09-05 12:39:32,077 44k INFO Saving model and optimizer state at iteration 8153 to ./logs\44k\D_171200.pth 2023-09-05 12:39:33,321 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_168800.pth 2023-09-05 12:39:33,358 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_168800.pth 2023-09-05 12:39:42,197 44k INFO ====> Epoch: 8153, cost 21.73 s 2023-09-05 12:39:56,789 44k INFO ====> Epoch: 8154, cost 14.59 s 2023-09-05 12:40:11,262 44k INFO ====> Epoch: 8155, cost 14.47 s 2023-09-05 12:40:25,727 44k INFO ====> Epoch: 8156, cost 14.46 s 2023-09-05 12:40:40,120 44k INFO ====> Epoch: 8157, cost 14.39 s 2023-09-05 12:40:54,929 44k INFO ====> Epoch: 8158, cost 14.81 s 2023-09-05 12:41:09,661 44k INFO ====> Epoch: 8159, cost 14.73 s 2023-09-05 12:41:23,915 44k INFO ====> Epoch: 8160, cost 14.25 s 2023-09-05 12:41:38,304 44k INFO ====> Epoch: 8161, cost 14.39 s 2023-09-05 12:41:51,639 44k INFO Train Epoch: 8162 [86%] 2023-09-05 12:41:51,639 44k INFO Losses: [2.481139898300171, 2.408310890197754, 9.371604919433594, 15.007736206054688, 0.6978448033332825], step: 171400, lr: 3.60526879568645e-05, reference_loss: 29.966636657714844 2023-09-05 12:41:53,059 44k INFO ====> Epoch: 8162, cost 14.75 s 2023-09-05 12:42:07,870 44k INFO ====> Epoch: 8163, cost 14.81 s 2023-09-05 12:42:22,598 44k INFO ====> Epoch: 8164, cost 14.73 s 2023-09-05 12:42:37,116 44k INFO ====> Epoch: 8165, cost 14.52 s 2023-09-05 12:42:51,622 44k INFO ====> Epoch: 8166, cost 14.51 s 2023-09-05 12:43:06,096 44k INFO ====> Epoch: 8167, cost 14.47 s 2023-09-05 12:43:20,582 44k INFO ====> Epoch: 8168, cost 14.49 s 2023-09-05 12:43:34,890 44k INFO ====> Epoch: 8169, cost 14.31 s 2023-09-05 12:43:49,555 44k INFO ====> Epoch: 8170, cost 14.66 s 2023-09-05 12:44:04,108 44k INFO ====> Epoch: 8171, cost 14.55 s 2023-09-05 12:44:10,479 44k INFO Train Epoch: 8172 [38%] 2023-09-05 12:44:10,480 44k INFO Losses: [1.9484657049179077, 2.8547773361206055, 10.174993515014648, 15.115487098693848, 0.5329990983009338], step: 171600, lr: 3.6007647438016627e-05, reference_loss: 30.62672233581543 2023-09-05 12:44:18,818 44k INFO ====> Epoch: 8172, cost 14.71 s 2023-09-05 12:44:33,400 44k INFO ====> Epoch: 8173, cost 14.58 s 2023-09-05 12:44:47,891 44k INFO ====> Epoch: 8174, cost 14.49 s 2023-09-05 12:45:02,357 44k INFO ====> Epoch: 8175, cost 14.47 s 2023-09-05 12:45:16,812 44k INFO ====> Epoch: 8176, cost 14.46 s 2023-09-05 12:45:31,059 44k INFO ====> Epoch: 8177, cost 14.25 s 2023-09-05 12:45:45,404 44k INFO ====> Epoch: 8178, cost 14.35 s 2023-09-05 12:45:59,960 44k INFO ====> Epoch: 8179, cost 14.56 s 2023-09-05 12:46:14,362 44k INFO ====> Epoch: 8180, cost 14.40 s 2023-09-05 12:46:28,525 44k INFO Train Epoch: 8181 [90%] 2023-09-05 12:46:28,525 44k INFO Losses: [2.166245937347412, 2.8100104331970215, 11.784013748168945, 16.86641502380371, 0.4308340549468994], step: 171800, lr: 3.5967159083044125e-05, reference_loss: 34.05752182006836 2023-09-05 12:46:29,221 44k INFO ====> Epoch: 8181, cost 14.86 s 2023-09-05 12:46:43,856 44k INFO ====> Epoch: 8182, cost 14.63 s 2023-09-05 12:46:58,552 44k INFO ====> Epoch: 8183, cost 14.70 s 2023-09-05 12:47:13,228 44k INFO ====> Epoch: 8184, cost 14.68 s 2023-09-05 12:47:27,780 44k INFO ====> Epoch: 8185, cost 14.55 s 2023-09-05 12:47:42,002 44k INFO ====> Epoch: 8186, cost 14.22 s 2023-09-05 12:47:56,769 44k INFO ====> Epoch: 8187, cost 14.77 s 2023-09-05 12:48:11,576 44k INFO ====> Epoch: 8188, cost 14.81 s 2023-09-05 12:48:26,056 44k INFO ====> Epoch: 8189, cost 14.48 s 2023-09-05 12:48:40,336 44k INFO ====> Epoch: 8190, cost 14.28 s 2023-09-05 12:48:47,471 44k INFO Train Epoch: 8191 [43%] 2023-09-05 12:48:47,472 44k INFO Losses: [1.8024243116378784, 3.2560582160949707, 10.306185722351074, 14.870312690734863, 0.6726959943771362], step: 172000, lr: 3.592222541517107e-05, reference_loss: 30.907676696777344 2023-09-05 12:48:53,050 44k INFO Saving model and optimizer state at iteration 8191 to ./logs\44k\G_172000.pth 2023-09-05 12:48:53,709 44k INFO Saving model and optimizer state at iteration 8191 to ./logs\44k\D_172000.pth 2023-09-05 12:48:55,303 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_169600.pth 2023-09-05 12:48:55,342 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_169600.pth 2023-09-05 12:49:02,884 44k INFO ====> Epoch: 8191, cost 22.55 s 2023-09-05 12:49:17,522 44k INFO ====> Epoch: 8192, cost 14.64 s 2023-09-05 12:49:31,789 44k INFO ====> Epoch: 8193, cost 14.27 s 2023-09-05 12:49:46,403 44k INFO ====> Epoch: 8194, cost 14.61 s 2023-09-05 12:50:00,961 44k INFO ====> Epoch: 8195, cost 14.56 s 2023-09-05 12:50:15,482 44k INFO ====> Epoch: 8196, cost 14.52 s 2023-09-05 12:50:30,101 44k INFO ====> Epoch: 8197, cost 14.62 s 2023-09-05 12:50:44,661 44k INFO ====> Epoch: 8198, cost 14.56 s 2023-09-05 12:50:59,238 44k INFO ====> Epoch: 8199, cost 14.58 s 2023-09-05 12:51:13,555 44k INFO Train Epoch: 8200 [95%] 2023-09-05 12:51:13,556 44k INFO Losses: [1.7640728950500488, 3.215146541595459, 11.68437385559082, 15.383159637451172, 0.5495533347129822], step: 172200, lr: 3.58818331119384e-05, reference_loss: 32.59630584716797 2023-09-05 12:51:13,822 44k INFO ====> Epoch: 8200, cost 14.58 s 2023-09-05 12:51:28,265 44k INFO ====> Epoch: 8201, cost 14.44 s 2023-09-05 12:51:42,688 44k INFO ====> Epoch: 8202, cost 14.42 s 2023-09-05 12:51:57,088 44k INFO ====> Epoch: 8203, cost 14.40 s 2023-09-05 12:52:11,740 44k INFO ====> Epoch: 8204, cost 14.65 s 2023-09-05 12:52:25,954 44k INFO ====> Epoch: 8205, cost 14.21 s 2023-09-05 12:52:40,464 44k INFO ====> Epoch: 8206, cost 14.51 s 2023-09-05 12:52:55,303 44k INFO ====> Epoch: 8207, cost 14.84 s 2023-09-05 12:53:09,896 44k INFO ====> Epoch: 8208, cost 14.59 s 2023-09-05 12:53:24,104 44k INFO ====> Epoch: 8209, cost 14.21 s 2023-09-05 12:53:31,776 44k INFO Train Epoch: 8210 [48%] 2023-09-05 12:53:31,776 44k INFO Losses: [2.259145736694336, 2.5418200492858887, 10.461579322814941, 16.544635772705078, 0.5877605676651001], step: 172400, lr: 3.58370060415544e-05, reference_loss: 32.39493942260742 2023-09-05 12:53:38,650 44k INFO ====> Epoch: 8210, cost 14.55 s 2023-09-05 12:53:53,544 44k INFO ====> Epoch: 8211, cost 14.89 s 2023-09-05 12:54:08,436 44k INFO ====> Epoch: 8212, cost 14.89 s 2023-09-05 12:54:23,185 44k INFO ====> Epoch: 8213, cost 14.75 s 2023-09-05 12:54:37,598 44k INFO ====> Epoch: 8214, cost 14.41 s 2023-09-05 12:54:52,066 44k INFO ====> Epoch: 8215, cost 14.47 s 2023-09-05 12:55:06,671 44k INFO ====> Epoch: 8216, cost 14.61 s 2023-09-05 12:55:21,470 44k INFO ====> Epoch: 8217, cost 14.80 s 2023-09-05 12:55:35,776 44k INFO ====> Epoch: 8218, cost 14.31 s 2023-09-05 12:55:50,441 44k INFO ====> Epoch: 8219, cost 14.66 s 2023-09-05 12:55:51,189 44k INFO Train Epoch: 8220 [0%] 2023-09-05 12:55:51,189 44k INFO Losses: [2.0376710891723633, 3.034247398376465, 8.881010055541992, 14.555207252502441, 0.6389859914779663], step: 172600, lr: 3.579223497349985e-05, reference_loss: 29.147123336791992 2023-09-05 12:56:05,294 44k INFO ====> Epoch: 8220, cost 14.85 s 2023-09-05 12:56:19,829 44k INFO ====> Epoch: 8221, cost 14.54 s 2023-09-05 12:56:34,447 44k INFO ====> Epoch: 8222, cost 14.62 s 2023-09-05 12:56:49,041 44k INFO ====> Epoch: 8223, cost 14.59 s 2023-09-05 12:57:03,671 44k INFO ====> Epoch: 8224, cost 14.63 s 2023-09-05 12:57:18,097 44k INFO ====> Epoch: 8225, cost 14.43 s 2023-09-05 12:57:32,722 44k INFO ====> Epoch: 8226, cost 14.62 s 2023-09-05 12:57:47,213 44k INFO ====> Epoch: 8227, cost 14.49 s 2023-09-05 12:58:01,806 44k INFO ====> Epoch: 8228, cost 14.59 s 2023-09-05 12:58:10,111 44k INFO Train Epoch: 8229 [52%] 2023-09-05 12:58:10,111 44k INFO Losses: [2.192164659500122, 2.7231240272521973, 11.823290824890137, 15.664777755737305, 0.6017196774482727], step: 172800, lr: 3.575198883641576e-05, reference_loss: 33.00507736206055 2023-09-05 12:58:15,633 44k INFO Saving model and optimizer state at iteration 8229 to ./logs\44k\G_172800.pth 2023-09-05 12:58:16,208 44k INFO Saving model and optimizer state at iteration 8229 to ./logs\44k\D_172800.pth 2023-09-05 12:58:17,482 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_170400.pth 2023-09-05 12:58:17,520 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_170400.pth 2023-09-05 12:58:23,588 44k INFO ====> Epoch: 8229, cost 21.78 s 2023-09-05 12:58:38,132 44k INFO ====> Epoch: 8230, cost 14.54 s 2023-09-05 12:58:52,717 44k INFO ====> Epoch: 8231, cost 14.59 s 2023-09-05 12:59:07,352 44k INFO ====> Epoch: 8232, cost 14.64 s 2023-09-05 12:59:22,011 44k INFO ====> Epoch: 8233, cost 14.66 s 2023-09-05 12:59:36,356 44k INFO ====> Epoch: 8234, cost 14.35 s 2023-09-05 12:59:51,096 44k INFO ====> Epoch: 8235, cost 14.74 s 2023-09-05 13:00:05,960 44k INFO ====> Epoch: 8236, cost 14.86 s 2023-09-05 13:00:20,600 44k INFO ====> Epoch: 8237, cost 14.64 s 2023-09-05 13:00:34,961 44k INFO ====> Epoch: 8238, cost 14.36 s 2023-09-05 13:00:36,379 44k INFO Train Epoch: 8239 [5%] 2023-09-05 13:00:36,379 44k INFO Losses: [2.1091606616973877, 2.8962020874023438, 8.632309913635254, 13.545836448669434, 0.6372153162956238], step: 173000, lr: 3.5707323980109845e-05, reference_loss: 27.820724487304688 2023-09-05 13:00:49,756 44k INFO ====> Epoch: 8239, cost 14.79 s 2023-09-05 13:01:04,429 44k INFO ====> Epoch: 8240, cost 14.67 s 2023-09-05 13:01:18,888 44k INFO ====> Epoch: 8241, cost 14.46 s 2023-09-05 13:01:33,408 44k INFO ====> Epoch: 8242, cost 14.52 s 2023-09-05 13:01:48,023 44k INFO ====> Epoch: 8243, cost 14.61 s 2023-09-05 13:02:02,631 44k INFO ====> Epoch: 8244, cost 14.61 s 2023-09-05 13:02:17,087 44k INFO ====> Epoch: 8245, cost 14.46 s 2023-09-05 13:02:31,714 44k INFO ====> Epoch: 8246, cost 14.63 s 2023-09-05 13:02:46,175 44k INFO ====> Epoch: 8247, cost 14.46 s 2023-09-05 13:02:55,494 44k INFO Train Epoch: 8248 [57%] 2023-09-05 13:02:55,495 44k INFO Losses: [2.369555950164795, 2.4210519790649414, 9.275727272033691, 16.066987991333008, 0.4871617257595062], step: 173200, lr: 3.566717332014482e-05, reference_loss: 30.620485305786133 2023-09-05 13:03:01,128 44k INFO ====> Epoch: 8248, cost 14.95 s 2023-09-05 13:03:15,585 44k INFO ====> Epoch: 8249, cost 14.46 s 2023-09-05 13:03:30,137 44k INFO ====> Epoch: 8250, cost 14.55 s 2023-09-05 13:03:44,613 44k INFO ====> Epoch: 8251, cost 14.48 s 2023-09-05 13:03:59,241 44k INFO ====> Epoch: 8252, cost 14.63 s 2023-09-05 13:04:13,836 44k INFO ====> Epoch: 8253, cost 14.59 s 2023-09-05 13:04:28,183 44k INFO ====> Epoch: 8254, cost 14.35 s 2023-09-05 13:04:42,954 44k INFO ====> Epoch: 8255, cost 14.77 s 2023-09-05 13:04:57,644 44k INFO ====> Epoch: 8256, cost 14.69 s 2023-09-05 13:05:12,201 44k INFO ====> Epoch: 8257, cost 14.56 s 2023-09-05 13:05:14,267 44k INFO Train Epoch: 8258 [10%] 2023-09-05 13:05:14,268 44k INFO Losses: [2.1473326683044434, 2.9313488006591797, 8.399922370910645, 14.758683204650879, 0.6054486036300659], step: 173400, lr: 3.56226144236182e-05, reference_loss: 28.84273338317871 2023-09-05 13:05:26,891 44k INFO ====> Epoch: 8258, cost 14.69 s 2023-09-05 13:05:41,469 44k INFO ====> Epoch: 8259, cost 14.58 s 2023-09-05 13:05:56,118 44k INFO ====> Epoch: 8260, cost 14.65 s 2023-09-05 13:06:10,842 44k INFO ====> Epoch: 8261, cost 14.72 s 2023-09-05 13:06:25,211 44k INFO ====> Epoch: 8262, cost 14.37 s 2023-09-05 13:06:39,616 44k INFO ====> Epoch: 8263, cost 14.41 s 2023-09-05 13:06:54,190 44k INFO ====> Epoch: 8264, cost 14.57 s 2023-09-05 13:07:08,765 44k INFO ====> Epoch: 8265, cost 14.57 s 2023-09-05 13:07:23,110 44k INFO ====> Epoch: 8266, cost 14.34 s 2023-09-05 13:07:32,744 44k INFO Train Epoch: 8267 [62%] 2023-09-05 13:07:32,744 44k INFO Losses: [2.0510363578796387, 3.079590320587158, 10.118231773376465, 14.481703758239746, 0.5575711727142334], step: 173600, lr: 3.558255901426899e-05, reference_loss: 30.28813362121582 2023-09-05 13:07:38,221 44k INFO Saving model and optimizer state at iteration 8267 to ./logs\44k\G_173600.pth 2023-09-05 13:07:38,851 44k INFO Saving model and optimizer state at iteration 8267 to ./logs\44k\D_173600.pth 2023-09-05 13:07:40,103 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_171200.pth 2023-09-05 13:07:40,139 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_171200.pth 2023-09-05 13:07:44,883 44k INFO ====> Epoch: 8267, cost 21.77 s 2023-09-05 13:07:59,557 44k INFO ====> Epoch: 8268, cost 14.67 s 2023-09-05 13:08:14,261 44k INFO ====> Epoch: 8269, cost 14.70 s 2023-09-05 13:08:28,794 44k INFO ====> Epoch: 8270, cost 14.53 s 2023-09-05 13:08:43,332 44k INFO ====> Epoch: 8271, cost 14.54 s 2023-09-05 13:08:57,956 44k INFO ====> Epoch: 8272, cost 14.62 s 2023-09-05 13:09:12,590 44k INFO ====> Epoch: 8273, cost 14.63 s 2023-09-05 13:09:27,032 44k INFO ====> Epoch: 8274, cost 14.44 s 2023-09-05 13:09:41,439 44k INFO ====> Epoch: 8275, cost 14.41 s 2023-09-05 13:09:56,027 44k INFO ====> Epoch: 8276, cost 14.59 s 2023-09-05 13:09:58,853 44k INFO Train Epoch: 8277 [14%] 2023-09-05 13:09:58,854 44k INFO Losses: [2.1383426189422607, 2.7348995208740234, 11.695845603942871, 16.683834075927734, 0.7161174416542053], step: 173800, lr: 3.5538105826150114e-05, reference_loss: 33.96903991699219 2023-09-05 13:10:10,888 44k INFO ====> Epoch: 8277, cost 14.86 s 2023-09-05 13:10:25,417 44k INFO ====> Epoch: 8278, cost 14.53 s 2023-09-05 13:10:39,909 44k INFO ====> Epoch: 8279, cost 14.49 s 2023-09-05 13:10:54,633 44k INFO ====> Epoch: 8280, cost 14.72 s 2023-09-05 13:11:09,438 44k INFO ====> Epoch: 8281, cost 14.81 s 2023-09-05 13:11:23,773 44k INFO ====> Epoch: 8282, cost 14.34 s 2023-09-05 13:11:38,080 44k INFO ====> Epoch: 8283, cost 14.31 s 2023-09-05 13:11:52,897 44k INFO ====> Epoch: 8284, cost 14.82 s 2023-09-05 13:12:07,589 44k INFO ====> Epoch: 8285, cost 14.69 s 2023-09-05 13:12:18,269 44k INFO Train Epoch: 8286 [67%] 2023-09-05 13:12:18,270 44k INFO Losses: [2.43291974067688, 2.533374309539795, 8.784996032714844, 15.38907527923584, 0.5535497665405273], step: 174000, lr: 3.549814544145083e-05, reference_loss: 29.69391632080078 2023-09-05 13:12:22,512 44k INFO ====> Epoch: 8286, cost 14.92 s 2023-09-05 13:12:37,267 44k INFO ====> Epoch: 8287, cost 14.75 s 2023-09-05 13:12:51,676 44k INFO ====> Epoch: 8288, cost 14.41 s 2023-09-05 13:13:06,088 44k INFO ====> Epoch: 8289, cost 14.41 s 2023-09-05 13:13:20,548 44k INFO ====> Epoch: 8290, cost 14.46 s 2023-09-05 13:13:34,835 44k INFO ====> Epoch: 8291, cost 14.29 s 2023-09-05 13:13:49,389 44k INFO ====> Epoch: 8292, cost 14.55 s 2023-09-05 13:14:04,001 44k INFO ====> Epoch: 8293, cost 14.61 s 2023-09-05 13:14:18,508 44k INFO ====> Epoch: 8294, cost 14.51 s 2023-09-05 13:14:32,906 44k INFO ====> Epoch: 8295, cost 14.40 s 2023-09-05 13:14:36,487 44k INFO Train Epoch: 8296 [19%] 2023-09-05 13:14:36,487 44k INFO Losses: [2.5235378742218018, 2.251060724258423, 7.46290922164917, 13.873607635498047, 0.5123530626296997], step: 174200, lr: 3.5453797710964464e-05, reference_loss: 26.62346839904785 2023-09-05 13:14:47,827 44k INFO ====> Epoch: 8296, cost 14.92 s 2023-09-05 13:15:02,409 44k INFO ====> Epoch: 8297, cost 14.58 s 2023-09-05 13:15:16,893 44k INFO ====> Epoch: 8298, cost 14.48 s 2023-09-05 13:15:31,298 44k INFO ====> Epoch: 8299, cost 14.40 s 2023-09-05 13:15:45,646 44k INFO ====> Epoch: 8300, cost 14.35 s 2023-09-05 13:16:00,188 44k INFO ====> Epoch: 8301, cost 14.54 s 2023-09-05 13:16:14,696 44k INFO ====> Epoch: 8302, cost 14.51 s 2023-09-05 13:16:29,189 44k INFO ====> Epoch: 8303, cost 14.49 s 2023-09-05 13:16:43,802 44k INFO ====> Epoch: 8304, cost 14.61 s 2023-09-05 13:16:55,101 44k INFO Train Epoch: 8305 [71%] 2023-09-05 13:16:55,102 44k INFO Losses: [2.1793370246887207, 2.6584393978118896, 10.413392066955566, 15.496277809143066, 0.6396536231040955], step: 174400, lr: 3.5413932125485286e-05, reference_loss: 31.387100219726562 2023-09-05 13:17:00,824 44k INFO Saving model and optimizer state at iteration 8305 to ./logs\44k\G_174400.pth 2023-09-05 13:17:01,523 44k INFO Saving model and optimizer state at iteration 8305 to ./logs\44k\D_174400.pth 2023-09-05 13:17:03,121 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_172000.pth 2023-09-05 13:17:03,155 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_172000.pth 2023-09-05 13:17:06,481 44k INFO ====> Epoch: 8305, cost 22.68 s 2023-09-05 13:17:20,978 44k INFO ====> Epoch: 8306, cost 14.50 s 2023-09-05 13:17:35,362 44k INFO ====> Epoch: 8307, cost 14.38 s 2023-09-05 13:17:49,972 44k INFO ====> Epoch: 8308, cost 14.61 s 2023-09-05 13:18:04,676 44k INFO ====> Epoch: 8309, cost 14.70 s 2023-09-05 13:18:19,218 44k INFO ====> Epoch: 8310, cost 14.54 s 2023-09-05 13:18:33,732 44k INFO ====> Epoch: 8311, cost 14.51 s 2023-09-05 13:18:48,242 44k INFO ====> Epoch: 8312, cost 14.51 s 2023-09-05 13:19:02,958 44k INFO ====> Epoch: 8313, cost 14.72 s 2023-09-05 13:19:17,311 44k INFO ====> Epoch: 8314, cost 14.35 s 2023-09-05 13:19:21,487 44k INFO Train Epoch: 8315 [24%] 2023-09-05 13:19:21,488 44k INFO Losses: [2.3940351009368896, 2.2023229598999023, 10.12443733215332, 15.468610763549805, 0.6402048468589783], step: 174600, lr: 3.536968960245111e-05, reference_loss: 30.829612731933594 2023-09-05 13:19:31,958 44k INFO ====> Epoch: 8315, cost 14.65 s 2023-09-05 13:19:46,595 44k INFO ====> Epoch: 8316, cost 14.64 s 2023-09-05 13:20:01,264 44k INFO ====> Epoch: 8317, cost 14.67 s 2023-09-05 13:20:15,783 44k INFO ====> Epoch: 8318, cost 14.52 s 2023-09-05 13:20:30,181 44k INFO ====> Epoch: 8319, cost 14.40 s 2023-09-05 13:20:44,794 44k INFO ====> Epoch: 8320, cost 14.61 s 2023-09-05 13:20:59,349 44k INFO ====> Epoch: 8321, cost 14.56 s 2023-09-05 13:21:13,927 44k INFO ====> Epoch: 8322, cost 14.58 s 2023-09-05 13:21:28,206 44k INFO ====> Epoch: 8323, cost 14.28 s 2023-09-05 13:21:40,217 44k INFO Train Epoch: 8324 [76%] 2023-09-05 13:21:40,218 44k INFO Losses: [2.1364834308624268, 2.7693302631378174, 10.488104820251465, 16.44498634338379, 0.6930454969406128], step: 174800, lr: 3.532991859129699e-05, reference_loss: 32.531951904296875 2023-09-05 13:21:43,027 44k INFO ====> Epoch: 8324, cost 14.82 s 2023-09-05 13:21:57,654 44k INFO ====> Epoch: 8325, cost 14.63 s 2023-09-05 13:22:12,399 44k INFO ====> Epoch: 8326, cost 14.75 s 2023-09-05 13:22:26,766 44k INFO ====> Epoch: 8327, cost 14.37 s 2023-09-05 13:22:41,276 44k INFO ====> Epoch: 8328, cost 14.51 s 2023-09-05 13:22:56,023 44k INFO ====> Epoch: 8329, cost 14.75 s 2023-09-05 13:23:10,774 44k INFO ====> Epoch: 8330, cost 14.75 s 2023-09-05 13:23:25,260 44k INFO ====> Epoch: 8331, cost 14.49 s 2023-09-05 13:23:39,678 44k INFO ====> Epoch: 8332, cost 14.42 s 2023-09-05 13:23:54,473 44k INFO ====> Epoch: 8333, cost 14.80 s 2023-09-05 13:23:59,398 44k INFO Train Epoch: 8334 [29%] 2023-09-05 13:23:59,398 44k INFO Losses: [2.4947397708892822, 2.210069179534912, 9.483445167541504, 13.371249198913574, 0.5872746109962463], step: 175000, lr: 3.528578102612822e-05, reference_loss: 28.146778106689453 2023-09-05 13:24:09,580 44k INFO ====> Epoch: 8334, cost 15.11 s 2023-09-05 13:24:24,054 44k INFO ====> Epoch: 8335, cost 14.47 s 2023-09-05 13:24:38,613 44k INFO ====> Epoch: 8336, cost 14.56 s 2023-09-05 13:24:53,187 44k INFO ====> Epoch: 8337, cost 14.57 s 2023-09-05 13:25:07,815 44k INFO ====> Epoch: 8338, cost 14.63 s 2023-09-05 13:25:22,252 44k INFO ====> Epoch: 8339, cost 14.44 s 2023-09-05 13:25:36,576 44k INFO ====> Epoch: 8340, cost 14.32 s 2023-09-05 13:25:51,052 44k INFO ====> Epoch: 8341, cost 14.48 s 2023-09-05 13:26:05,767 44k INFO ====> Epoch: 8342, cost 14.71 s 2023-09-05 13:26:18,487 44k INFO Train Epoch: 8343 [81%] 2023-09-05 13:26:18,488 44k INFO Losses: [2.1195871829986572, 2.8655989170074463, 10.275381088256836, 14.084970474243164, 0.6599278450012207], step: 175200, lr: 3.5246104364937646e-05, reference_loss: 30.005464553833008 2023-09-05 13:26:24,047 44k INFO Saving model and optimizer state at iteration 8343 to ./logs\44k\G_175200.pth 2023-09-05 13:26:24,666 44k INFO Saving model and optimizer state at iteration 8343 to ./logs\44k\D_175200.pth 2023-09-05 13:26:25,895 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_172800.pth 2023-09-05 13:26:25,935 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_172800.pth 2023-09-05 13:26:27,787 44k INFO ====> Epoch: 8343, cost 22.02 s 2023-09-05 13:26:42,343 44k INFO ====> Epoch: 8344, cost 14.56 s 2023-09-05 13:26:57,000 44k INFO ====> Epoch: 8345, cost 14.66 s 2023-09-05 13:27:11,667 44k INFO ====> Epoch: 8346, cost 14.67 s 2023-09-05 13:27:26,064 44k INFO ====> Epoch: 8347, cost 14.40 s 2023-09-05 13:27:40,543 44k INFO ====> Epoch: 8348, cost 14.48 s 2023-09-05 13:27:55,093 44k INFO ====> Epoch: 8349, cost 14.55 s 2023-09-05 13:28:09,772 44k INFO ====> Epoch: 8350, cost 14.68 s 2023-09-05 13:28:24,174 44k INFO ====> Epoch: 8351, cost 14.40 s 2023-09-05 13:28:38,581 44k INFO ====> Epoch: 8352, cost 14.41 s 2023-09-05 13:28:44,147 44k INFO Train Epoch: 8353 [33%] 2023-09-05 13:28:44,147 44k INFO Losses: [2.1506667137145996, 2.7529091835021973, 9.205828666687012, 14.322526931762695, 0.6780940890312195], step: 175400, lr: 3.5202071508639585e-05, reference_loss: 29.11002540588379 2023-09-05 13:28:53,328 44k INFO ====> Epoch: 8353, cost 14.75 s 2023-09-05 13:29:07,892 44k INFO ====> Epoch: 8354, cost 14.56 s 2023-09-05 13:29:22,302 44k INFO ====> Epoch: 8355, cost 14.41 s 2023-09-05 13:29:36,785 44k INFO ====> Epoch: 8356, cost 14.48 s 2023-09-05 13:29:51,392 44k INFO ====> Epoch: 8357, cost 14.61 s 2023-09-05 13:30:06,160 44k INFO ====> Epoch: 8358, cost 14.77 s 2023-09-05 13:30:20,769 44k INFO ====> Epoch: 8359, cost 14.61 s 2023-09-05 13:30:35,137 44k INFO ====> Epoch: 8360, cost 14.37 s 2023-09-05 13:30:49,623 44k INFO ====> Epoch: 8361, cost 14.49 s 2023-09-05 13:31:03,080 44k INFO Train Epoch: 8362 [86%] 2023-09-05 13:31:03,080 44k INFO Losses: [2.257190227508545, 2.893336772918701, 9.877387046813965, 15.527289390563965, 0.5781210660934448], step: 175600, lr: 3.5162488973583317e-05, reference_loss: 31.133323669433594 2023-09-05 13:31:04,467 44k INFO ====> Epoch: 8362, cost 14.84 s 2023-09-05 13:31:19,005 44k INFO ====> Epoch: 8363, cost 14.54 s 2023-09-05 13:31:33,454 44k INFO ====> Epoch: 8364, cost 14.45 s 2023-09-05 13:31:48,047 44k INFO ====> Epoch: 8365, cost 14.59 s 2023-09-05 13:32:02,674 44k INFO ====> Epoch: 8366, cost 14.63 s 2023-09-05 13:32:17,123 44k INFO ====> Epoch: 8367, cost 14.45 s 2023-09-05 13:32:31,623 44k INFO ====> Epoch: 8368, cost 14.50 s 2023-09-05 13:32:46,173 44k INFO ====> Epoch: 8369, cost 14.55 s 2023-09-05 13:33:00,643 44k INFO ====> Epoch: 8370, cost 14.47 s 2023-09-05 13:33:15,073 44k INFO ====> Epoch: 8371, cost 14.43 s 2023-09-05 13:33:21,399 44k INFO Train Epoch: 8372 [38%] 2023-09-05 13:33:21,400 44k INFO Losses: [2.010209798812866, 2.885777711868286, 9.930574417114258, 15.117137908935547, 0.5308272242546082], step: 175800, lr: 3.5118560577751975e-05, reference_loss: 30.474525451660156 2023-09-05 13:33:29,828 44k INFO ====> Epoch: 8372, cost 14.76 s 2023-09-05 13:33:44,209 44k INFO ====> Epoch: 8373, cost 14.38 s 2023-09-05 13:33:58,696 44k INFO ====> Epoch: 8374, cost 14.49 s 2023-09-05 13:34:13,141 44k INFO ====> Epoch: 8375, cost 14.44 s 2023-09-05 13:34:27,432 44k INFO ====> Epoch: 8376, cost 14.29 s 2023-09-05 13:34:42,185 44k INFO ====> Epoch: 8377, cost 14.75 s 2023-09-05 13:34:57,069 44k INFO ====> Epoch: 8378, cost 14.88 s 2023-09-05 13:35:11,759 44k INFO ====> Epoch: 8379, cost 14.69 s 2023-09-05 13:35:26,210 44k INFO ====> Epoch: 8380, cost 14.45 s 2023-09-05 13:35:40,148 44k INFO Train Epoch: 8381 [90%] 2023-09-05 13:35:40,148 44k INFO Losses: [1.9566923379898071, 2.96561336517334, 9.772664070129395, 16.193355560302734, 0.7216029763221741], step: 176000, lr: 3.507907194553175e-05, reference_loss: 31.609928131103516 2023-09-05 13:35:45,660 44k INFO Saving model and optimizer state at iteration 8381 to ./logs\44k\G_176000.pth 2023-09-05 13:35:46,276 44k INFO Saving model and optimizer state at iteration 8381 to ./logs\44k\D_176000.pth 2023-09-05 13:35:47,870 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_173600.pth 2023-09-05 13:35:47,905 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_173600.pth 2023-09-05 13:35:48,357 44k INFO ====> Epoch: 8381, cost 22.15 s 2023-09-05 13:36:03,276 44k INFO ====> Epoch: 8382, cost 14.92 s 2023-09-05 13:36:17,893 44k INFO ====> Epoch: 8383, cost 14.62 s 2023-09-05 13:36:32,595 44k INFO ====> Epoch: 8384, cost 14.70 s 2023-09-05 13:36:47,274 44k INFO ====> Epoch: 8385, cost 14.68 s 2023-09-05 13:37:01,820 44k INFO ====> Epoch: 8386, cost 14.55 s 2023-09-05 13:37:16,450 44k INFO ====> Epoch: 8387, cost 14.63 s 2023-09-05 13:37:30,891 44k INFO ====> Epoch: 8388, cost 14.44 s 2023-09-05 13:37:45,339 44k INFO ====> Epoch: 8389, cost 14.45 s 2023-09-05 13:37:59,936 44k INFO ====> Epoch: 8390, cost 14.60 s 2023-09-05 13:38:06,961 44k INFO Train Epoch: 8391 [43%] 2023-09-05 13:38:06,962 44k INFO Losses: [2.1409740447998047, 2.740983724594116, 8.92721176147461, 15.261817932128906, 0.6418808698654175], step: 176200, lr: 3.5035247762352416e-05, reference_loss: 29.712867736816406 2023-09-05 13:38:14,692 44k INFO ====> Epoch: 8391, cost 14.76 s 2023-09-05 13:38:29,237 44k INFO ====> Epoch: 8392, cost 14.55 s 2023-09-05 13:38:43,740 44k INFO ====> Epoch: 8393, cost 14.50 s 2023-09-05 13:38:58,147 44k INFO ====> Epoch: 8394, cost 14.41 s 2023-09-05 13:39:12,759 44k INFO ====> Epoch: 8395, cost 14.61 s 2023-09-05 13:39:27,080 44k INFO ====> Epoch: 8396, cost 14.32 s 2023-09-05 13:39:41,497 44k INFO ====> Epoch: 8397, cost 14.42 s 2023-09-05 13:39:56,083 44k INFO ====> Epoch: 8398, cost 14.59 s 2023-09-05 13:40:10,753 44k INFO ====> Epoch: 8399, cost 14.67 s 2023-09-05 13:40:25,227 44k INFO Train Epoch: 8400 [95%] 2023-09-05 13:40:25,227 44k INFO Losses: [1.8065255880355835, 3.25253963470459, 10.097180366516113, 15.275524139404297, 0.30231907963752747], step: 176400, lr: 3.499585281019974e-05, reference_loss: 30.734088897705078 2023-09-05 13:40:25,493 44k INFO ====> Epoch: 8400, cost 14.74 s 2023-09-05 13:40:39,961 44k INFO ====> Epoch: 8401, cost 14.47 s 2023-09-05 13:40:54,415 44k INFO ====> Epoch: 8402, cost 14.45 s 2023-09-05 13:41:09,150 44k INFO ====> Epoch: 8403, cost 14.74 s 2023-09-05 13:41:23,581 44k INFO ====> Epoch: 8404, cost 14.43 s 2023-09-05 13:41:37,959 44k INFO ====> Epoch: 8405, cost 14.38 s 2023-09-05 13:41:52,688 44k INFO ====> Epoch: 8406, cost 14.73 s 2023-09-05 13:42:07,358 44k INFO ====> Epoch: 8407, cost 14.67 s 2023-09-05 13:42:21,878 44k INFO ====> Epoch: 8408, cost 14.52 s 2023-09-05 13:42:36,237 44k INFO ====> Epoch: 8409, cost 14.36 s 2023-09-05 13:42:44,112 44k INFO Train Epoch: 8410 [48%] 2023-09-05 13:42:44,112 44k INFO Losses: [1.984722375869751, 3.0500547885894775, 10.692849159240723, 15.728799819946289, 0.5593437552452087], step: 176600, lr: 3.495213259244562e-05, reference_loss: 32.015769958496094 2023-09-05 13:42:51,223 44k INFO ====> Epoch: 8410, cost 14.99 s 2023-09-05 13:43:05,894 44k INFO ====> Epoch: 8411, cost 14.67 s 2023-09-05 13:43:20,455 44k INFO ====> Epoch: 8412, cost 14.56 s 2023-09-05 13:43:34,752 44k INFO ====> Epoch: 8413, cost 14.30 s 2023-09-05 13:43:49,259 44k INFO ====> Epoch: 8414, cost 14.51 s 2023-09-05 13:44:03,784 44k INFO ====> Epoch: 8415, cost 14.52 s 2023-09-05 13:44:18,302 44k INFO ====> Epoch: 8416, cost 14.52 s 2023-09-05 13:44:32,783 44k INFO ====> Epoch: 8417, cost 14.48 s 2023-09-05 13:44:47,320 44k INFO ====> Epoch: 8418, cost 14.54 s 2023-09-05 13:45:01,907 44k INFO ====> Epoch: 8419, cost 14.59 s 2023-09-05 13:45:02,602 44k INFO Train Epoch: 8420 [0%] 2023-09-05 13:45:02,602 44k INFO Losses: [2.223106861114502, 2.652245283126831, 9.941147804260254, 15.180990219116211, 0.5939034819602966], step: 176800, lr: 3.490846699423316e-05, reference_loss: 30.591392517089844 2023-09-05 13:45:08,077 44k INFO Saving model and optimizer state at iteration 8420 to ./logs\44k\G_176800.pth 2023-09-05 13:45:08,718 44k INFO Saving model and optimizer state at iteration 8420 to ./logs\44k\D_176800.pth 2023-09-05 13:45:10,328 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_174400.pth 2023-09-05 13:45:10,366 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_174400.pth 2023-09-05 13:45:23,915 44k INFO ====> Epoch: 8420, cost 22.01 s 2023-09-05 13:45:38,266 44k INFO ====> Epoch: 8421, cost 14.35 s 2023-09-05 13:45:52,846 44k INFO ====> Epoch: 8422, cost 14.58 s 2023-09-05 13:46:07,451 44k INFO ====> Epoch: 8423, cost 14.60 s 2023-09-05 13:46:21,970 44k INFO ====> Epoch: 8424, cost 14.52 s 2023-09-05 13:46:36,389 44k INFO ====> Epoch: 8425, cost 14.42 s 2023-09-05 13:46:50,913 44k INFO ====> Epoch: 8426, cost 14.52 s 2023-09-05 13:47:05,681 44k INFO ====> Epoch: 8427, cost 14.77 s 2023-09-05 13:47:20,136 44k INFO ====> Epoch: 8428, cost 14.46 s 2023-09-05 13:47:28,566 44k INFO Train Epoch: 8429 [52%] 2023-09-05 13:47:28,566 44k INFO Losses: [2.3569891452789307, 2.462796211242676, 8.196606636047363, 15.275686264038086, 0.6942118406295776], step: 177000, lr: 3.486921459915122e-05, reference_loss: 28.986289978027344 2023-09-05 13:47:34,960 44k INFO ====> Epoch: 8429, cost 14.82 s 2023-09-05 13:47:49,473 44k INFO ====> Epoch: 8430, cost 14.51 s 2023-09-05 13:48:04,253 44k INFO ====> Epoch: 8431, cost 14.78 s 2023-09-05 13:48:18,811 44k INFO ====> Epoch: 8432, cost 14.56 s 2023-09-05 13:48:33,282 44k INFO ====> Epoch: 8433, cost 14.47 s 2023-09-05 13:48:47,776 44k INFO ====> Epoch: 8434, cost 14.49 s 2023-09-05 13:49:02,495 44k INFO ====> Epoch: 8435, cost 14.72 s 2023-09-05 13:49:17,128 44k INFO ====> Epoch: 8436, cost 14.63 s 2023-09-05 13:49:31,503 44k INFO ====> Epoch: 8437, cost 14.37 s 2023-09-05 13:49:46,043 44k INFO ====> Epoch: 8438, cost 14.54 s 2023-09-05 13:49:47,465 44k INFO Train Epoch: 8439 [5%] 2023-09-05 13:49:47,465 44k INFO Losses: [2.118119716644287, 2.9495809078216553, 11.852937698364258, 13.993274688720703, 0.6132065057754517], step: 177200, lr: 3.48256525901481e-05, reference_loss: 31.52712059020996 2023-09-05 13:50:00,984 44k INFO ====> Epoch: 8439, cost 14.94 s 2023-09-05 13:50:15,557 44k INFO ====> Epoch: 8440, cost 14.57 s 2023-09-05 13:50:30,096 44k INFO ====> Epoch: 8441, cost 14.54 s 2023-09-05 13:50:44,765 44k INFO ====> Epoch: 8442, cost 14.67 s 2023-09-05 13:50:59,420 44k INFO ====> Epoch: 8443, cost 14.66 s 2023-09-05 13:51:13,872 44k INFO ====> Epoch: 8444, cost 14.45 s 2023-09-05 13:51:28,174 44k INFO ====> Epoch: 8445, cost 14.30 s 2023-09-05 13:51:42,523 44k INFO ====> Epoch: 8446, cost 14.35 s 2023-09-05 13:51:57,099 44k INFO ====> Epoch: 8447, cost 14.58 s 2023-09-05 13:52:06,374 44k INFO Train Epoch: 8448 [57%] 2023-09-05 13:52:06,374 44k INFO Losses: [2.0809178352355957, 2.8996200561523438, 11.833026885986328, 16.069263458251953, 0.48803451657295227], step: 177400, lr: 3.478649331470124e-05, reference_loss: 33.3708610534668 2023-09-05 13:52:12,010 44k INFO ====> Epoch: 8448, cost 14.91 s 2023-09-05 13:52:26,638 44k INFO ====> Epoch: 8449, cost 14.63 s 2023-09-05 13:52:41,145 44k INFO ====> Epoch: 8450, cost 14.51 s 2023-09-05 13:52:55,725 44k INFO ====> Epoch: 8451, cost 14.58 s 2023-09-05 13:53:10,359 44k INFO ====> Epoch: 8452, cost 14.63 s 2023-09-05 13:53:24,837 44k INFO ====> Epoch: 8453, cost 14.48 s 2023-09-05 13:53:39,238 44k INFO ====> Epoch: 8454, cost 14.40 s 2023-09-05 13:53:54,084 44k INFO ====> Epoch: 8455, cost 14.85 s 2023-09-05 13:54:08,734 44k INFO ====> Epoch: 8456, cost 14.65 s 2023-09-05 13:54:23,418 44k INFO ====> Epoch: 8457, cost 14.68 s 2023-09-05 13:54:25,530 44k INFO Train Epoch: 8458 [10%] 2023-09-05 13:54:25,530 44k INFO Losses: [2.1666064262390137, 2.711332321166992, 10.683123588562012, 14.198148727416992, 0.6960861682891846], step: 177600, lr: 3.474303464915966e-05, reference_loss: 30.455297470092773 2023-09-05 13:54:31,099 44k INFO Saving model and optimizer state at iteration 8458 to ./logs\44k\G_177600.pth 2023-09-05 13:54:31,694 44k INFO Saving model and optimizer state at iteration 8458 to ./logs\44k\D_177600.pth 2023-09-05 13:54:32,830 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_175200.pth 2023-09-05 13:54:32,863 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_175200.pth 2023-09-05 13:54:45,396 44k INFO ====> Epoch: 8458, cost 21.98 s 2023-09-05 13:54:59,981 44k INFO ====> Epoch: 8459, cost 14.59 s 2023-09-05 13:55:14,486 44k INFO ====> Epoch: 8460, cost 14.50 s 2023-09-05 13:55:28,912 44k INFO ====> Epoch: 8461, cost 14.43 s 2023-09-05 13:55:43,342 44k INFO ====> Epoch: 8462, cost 14.43 s 2023-09-05 13:55:58,178 44k INFO ====> Epoch: 8463, cost 14.84 s 2023-09-05 13:56:12,788 44k INFO ====> Epoch: 8464, cost 14.61 s 2023-09-05 13:56:27,324 44k INFO ====> Epoch: 8465, cost 14.54 s 2023-09-05 13:56:41,913 44k INFO ====> Epoch: 8466, cost 14.59 s 2023-09-05 13:56:51,899 44k INFO Train Epoch: 8467 [62%] 2023-09-05 13:56:51,899 44k INFO Losses: [2.291287422180176, 2.7078728675842285, 8.429481506347656, 15.880241394042969, 0.5803349018096924], step: 177800, lr: 3.470396827243738e-05, reference_loss: 29.889219284057617 2023-09-05 13:56:56,844 44k INFO ====> Epoch: 8467, cost 14.93 s 2023-09-05 13:57:11,545 44k INFO ====> Epoch: 8468, cost 14.70 s 2023-09-05 13:57:25,704 44k INFO ====> Epoch: 8469, cost 14.16 s 2023-09-05 13:57:40,169 44k INFO ====> Epoch: 8470, cost 14.47 s 2023-09-05 13:57:54,836 44k INFO ====> Epoch: 8471, cost 14.67 s 2023-09-05 13:58:09,763 44k INFO ====> Epoch: 8472, cost 14.93 s 2023-09-05 13:58:24,184 44k INFO ====> Epoch: 8473, cost 14.42 s 2023-09-05 13:58:38,559 44k INFO ====> Epoch: 8474, cost 14.37 s 2023-09-05 13:58:53,151 44k INFO ====> Epoch: 8475, cost 14.59 s 2023-09-05 13:59:07,870 44k INFO ====> Epoch: 8476, cost 14.72 s 2023-09-05 13:59:10,726 44k INFO Train Epoch: 8477 [14%] 2023-09-05 13:59:10,726 44k INFO Losses: [2.3099544048309326, 2.8019957542419434, 9.062005043029785, 15.730695724487305, 0.5489979386329651], step: 178000, lr: 3.466061270519255e-05, reference_loss: 30.453649520874023 2023-09-05 13:59:22,537 44k INFO ====> Epoch: 8477, cost 14.67 s 2023-09-05 13:59:37,048 44k INFO ====> Epoch: 8478, cost 14.51 s 2023-09-05 13:59:51,784 44k INFO ====> Epoch: 8479, cost 14.74 s 2023-09-05 14:00:06,687 44k INFO ====> Epoch: 8480, cost 14.90 s 2023-09-05 14:00:21,194 44k INFO ====> Epoch: 8481, cost 14.51 s 2023-09-05 14:00:35,790 44k INFO ====> Epoch: 8482, cost 14.60 s 2023-09-05 14:00:50,386 44k INFO ====> Epoch: 8483, cost 14.60 s 2023-09-05 14:01:05,041 44k INFO ====> Epoch: 8484, cost 14.66 s 2023-09-05 14:01:19,547 44k INFO ====> Epoch: 8485, cost 14.51 s 2023-09-05 14:01:29,984 44k INFO Train Epoch: 8486 [67%] 2023-09-05 14:01:29,985 44k INFO Losses: [1.7577195167541504, 3.2382965087890625, 8.357919692993164, 12.711356163024902, 0.5234512686729431], step: 178200, lr: 3.46216390068084e-05, reference_loss: 26.588743209838867 2023-09-05 14:01:34,172 44k INFO ====> Epoch: 8486, cost 14.62 s 2023-09-05 14:01:48,742 44k INFO ====> Epoch: 8487, cost 14.57 s 2023-09-05 14:02:03,531 44k INFO ====> Epoch: 8488, cost 14.79 s 2023-09-05 14:02:18,188 44k INFO ====> Epoch: 8489, cost 14.66 s 2023-09-05 14:02:32,653 44k INFO ====> Epoch: 8490, cost 14.47 s 2023-09-05 14:02:47,281 44k INFO ====> Epoch: 8491, cost 14.63 s 2023-09-05 14:03:01,932 44k INFO ====> Epoch: 8492, cost 14.65 s 2023-09-05 14:03:16,548 44k INFO ====> Epoch: 8493, cost 14.62 s 2023-09-05 14:03:31,029 44k INFO ====> Epoch: 8494, cost 14.48 s 2023-09-05 14:03:45,530 44k INFO ====> Epoch: 8495, cost 14.50 s 2023-09-05 14:03:49,048 44k INFO Train Epoch: 8496 [19%] 2023-09-05 14:03:49,048 44k INFO Losses: [2.100796699523926, 2.920851707458496, 10.76782512664795, 14.798239707946777, 0.7051165103912354], step: 178400, lr: 3.457838629327713e-05, reference_loss: 31.292829513549805 2023-09-05 14:03:54,534 44k INFO Saving model and optimizer state at iteration 8496 to ./logs\44k\G_178400.pth 2023-09-05 14:03:55,191 44k INFO Saving model and optimizer state at iteration 8496 to ./logs\44k\D_178400.pth 2023-09-05 14:03:56,411 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_176000.pth 2023-09-05 14:03:56,450 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_176000.pth 2023-09-05 14:04:07,534 44k INFO ====> Epoch: 8496, cost 22.00 s 2023-09-05 14:04:22,095 44k INFO ====> Epoch: 8497, cost 14.56 s 2023-09-05 14:04:36,587 44k INFO ====> Epoch: 8498, cost 14.49 s 2023-09-05 14:04:51,212 44k INFO ====> Epoch: 8499, cost 14.63 s 2023-09-05 14:05:05,858 44k INFO ====> Epoch: 8500, cost 14.65 s 2023-09-05 14:05:20,428 44k INFO ====> Epoch: 8501, cost 14.57 s 2023-09-05 14:05:34,702 44k INFO ====> Epoch: 8502, cost 14.27 s 2023-09-05 14:05:49,485 44k INFO ====> Epoch: 8503, cost 14.78 s 2023-09-05 14:06:04,222 44k INFO ====> Epoch: 8504, cost 14.74 s 2023-09-05 14:06:15,643 44k INFO Train Epoch: 8505 [71%] 2023-09-05 14:06:15,643 44k INFO Losses: [2.2538278102874756, 2.360375165939331, 8.272904396057129, 12.552024841308594, 0.7443541288375854], step: 178600, lr: 3.453950505336751e-05, reference_loss: 26.183486938476562 2023-09-05 14:06:19,120 44k INFO ====> Epoch: 8505, cost 14.90 s 2023-09-05 14:06:33,567 44k INFO ====> Epoch: 8506, cost 14.45 s 2023-09-05 14:06:48,191 44k INFO ====> Epoch: 8507, cost 14.62 s 2023-09-05 14:07:02,822 44k INFO ====> Epoch: 8508, cost 14.63 s 2023-09-05 14:07:17,413 44k INFO ====> Epoch: 8509, cost 14.59 s 2023-09-05 14:07:31,736 44k INFO ====> Epoch: 8510, cost 14.32 s 2023-09-05 14:07:46,163 44k INFO ====> Epoch: 8511, cost 14.43 s 2023-09-05 14:08:00,869 44k INFO ====> Epoch: 8512, cost 14.71 s 2023-09-05 14:08:15,491 44k INFO ====> Epoch: 8513, cost 14.62 s 2023-09-05 14:08:29,976 44k INFO ====> Epoch: 8514, cost 14.49 s 2023-09-05 14:08:34,245 44k INFO Train Epoch: 8515 [24%] 2023-09-05 14:08:34,246 44k INFO Losses: [2.4679501056671143, 2.3753273487091064, 9.045675277709961, 14.428574562072754, 0.5110101103782654], step: 178800, lr: 3.4496354949546856e-05, reference_loss: 28.828536987304688 2023-09-05 14:08:44,893 44k INFO ====> Epoch: 8515, cost 14.92 s 2023-09-05 14:08:59,508 44k INFO ====> Epoch: 8516, cost 14.62 s 2023-09-05 14:09:13,991 44k INFO ====> Epoch: 8517, cost 14.48 s 2023-09-05 14:09:28,228 44k INFO ====> Epoch: 8518, cost 14.24 s 2023-09-05 14:09:42,462 44k INFO ====> Epoch: 8519, cost 14.23 s 2023-09-05 14:09:57,097 44k INFO ====> Epoch: 8520, cost 14.64 s 2023-09-05 14:10:11,663 44k INFO ====> Epoch: 8521, cost 14.57 s 2023-09-05 14:10:26,063 44k INFO ====> Epoch: 8522, cost 14.40 s 2023-09-05 14:10:40,492 44k INFO ====> Epoch: 8523, cost 14.43 s 2023-09-05 14:10:52,456 44k INFO Train Epoch: 8524 [76%] 2023-09-05 14:10:52,457 44k INFO Losses: [2.4306414127349854, 2.3679866790771484, 8.932649612426758, 13.288461685180664, 0.45398613810539246], step: 179000, lr: 3.4457565948769774e-05, reference_loss: 27.473724365234375 2023-09-05 14:10:55,329 44k INFO ====> Epoch: 8524, cost 14.84 s 2023-09-05 14:11:10,010 44k INFO ====> Epoch: 8525, cost 14.68 s 2023-09-05 14:11:24,305 44k INFO ====> Epoch: 8526, cost 14.29 s 2023-09-05 14:11:38,725 44k INFO ====> Epoch: 8527, cost 14.42 s 2023-09-05 14:11:53,388 44k INFO ====> Epoch: 8528, cost 14.66 s 2023-09-05 14:12:07,936 44k INFO ====> Epoch: 8529, cost 14.55 s 2023-09-05 14:12:22,514 44k INFO ====> Epoch: 8530, cost 14.58 s 2023-09-05 14:12:36,979 44k INFO ====> Epoch: 8531, cost 14.47 s 2023-09-05 14:12:51,468 44k INFO ====> Epoch: 8532, cost 14.49 s 2023-09-05 14:13:06,233 44k INFO ====> Epoch: 8533, cost 14.77 s 2023-09-05 14:13:11,239 44k INFO Train Epoch: 8534 [29%] 2023-09-05 14:13:11,239 44k INFO Losses: [2.082688808441162, 2.830157995223999, 12.333963394165039, 16.02035140991211, 0.7642242312431335], step: 179200, lr: 3.441451821123563e-05, reference_loss: 34.03138732910156 2023-09-05 14:13:16,814 44k INFO Saving model and optimizer state at iteration 8534 to ./logs\44k\G_179200.pth 2023-09-05 14:13:17,490 44k INFO Saving model and optimizer state at iteration 8534 to ./logs\44k\D_179200.pth 2023-09-05 14:13:18,569 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_176800.pth 2023-09-05 14:13:18,607 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_176800.pth 2023-09-05 14:13:28,189 44k INFO ====> Epoch: 8534, cost 21.96 s 2023-09-05 14:13:42,567 44k INFO ====> Epoch: 8535, cost 14.38 s 2023-09-05 14:13:57,206 44k INFO ====> Epoch: 8536, cost 14.64 s 2023-09-05 14:14:11,673 44k INFO ====> Epoch: 8537, cost 14.47 s 2023-09-05 14:14:26,181 44k INFO ====> Epoch: 8538, cost 14.51 s 2023-09-05 14:14:40,690 44k INFO ====> Epoch: 8539, cost 14.51 s 2023-09-05 14:14:55,039 44k INFO ====> Epoch: 8540, cost 14.35 s 2023-09-05 14:15:09,596 44k INFO ====> Epoch: 8541, cost 14.56 s 2023-09-05 14:15:24,008 44k INFO ====> Epoch: 8542, cost 14.41 s 2023-09-05 14:15:36,700 44k INFO Train Epoch: 8543 [81%] 2023-09-05 14:15:36,700 44k INFO Losses: [2.1656718254089355, 2.6581737995147705, 9.221709251403809, 15.955765724182129, 0.6346988677978516], step: 179400, lr: 3.43758212307694e-05, reference_loss: 30.636018753051758 2023-09-05 14:15:38,767 44k INFO ====> Epoch: 8543, cost 14.76 s 2023-09-05 14:15:53,227 44k INFO ====> Epoch: 8544, cost 14.46 s 2023-09-05 14:16:07,966 44k INFO ====> Epoch: 8545, cost 14.74 s 2023-09-05 14:16:22,412 44k INFO ====> Epoch: 8546, cost 14.45 s 2023-09-05 14:16:36,908 44k INFO ====> Epoch: 8547, cost 14.50 s 2023-09-05 14:16:51,460 44k INFO ====> Epoch: 8548, cost 14.55 s 2023-09-05 14:17:06,014 44k INFO ====> Epoch: 8549, cost 14.55 s 2023-09-05 14:17:20,534 44k INFO ====> Epoch: 8550, cost 14.52 s 2023-09-05 14:17:34,942 44k INFO ====> Epoch: 8551, cost 14.41 s 2023-09-05 14:17:49,605 44k INFO ====> Epoch: 8552, cost 14.66 s 2023-09-05 14:17:55,324 44k INFO Train Epoch: 8553 [33%] 2023-09-05 14:17:55,324 44k INFO Losses: [2.2709741592407227, 2.4139626026153564, 10.922656059265137, 15.994376182556152, 0.5943562388420105], step: 179600, lr: 3.4332875616675144e-05, reference_loss: 32.19632339477539 2023-09-05 14:18:04,676 44k INFO ====> Epoch: 8553, cost 15.07 s 2023-09-05 14:18:19,373 44k INFO ====> Epoch: 8554, cost 14.70 s 2023-09-05 14:18:33,841 44k INFO ====> Epoch: 8555, cost 14.47 s 2023-09-05 14:18:48,485 44k INFO ====> Epoch: 8556, cost 14.64 s 2023-09-05 14:19:03,019 44k INFO ====> Epoch: 8557, cost 14.53 s 2023-09-05 14:19:17,497 44k INFO ====> Epoch: 8558, cost 14.48 s 2023-09-05 14:19:31,764 44k INFO ====> Epoch: 8559, cost 14.27 s 2023-09-05 14:19:46,217 44k INFO ====> Epoch: 8560, cost 14.45 s 2023-09-05 14:20:00,913 44k INFO ====> Epoch: 8561, cost 14.70 s 2023-09-05 14:20:14,434 44k INFO Train Epoch: 8562 [86%] 2023-09-05 14:20:14,434 44k INFO Losses: [2.1908793449401855, 2.856921434402466, 10.333991050720215, 15.053999900817871, 0.6782189011573792], step: 179800, lr: 3.4294270438217224e-05, reference_loss: 31.114009857177734 2023-09-05 14:20:15,828 44k INFO ====> Epoch: 8562, cost 14.92 s 2023-09-05 14:20:30,372 44k INFO ====> Epoch: 8563, cost 14.54 s 2023-09-05 14:20:44,719 44k INFO ====> Epoch: 8564, cost 14.35 s 2023-09-05 14:20:59,430 44k INFO ====> Epoch: 8565, cost 14.71 s 2023-09-05 14:21:13,842 44k INFO ====> Epoch: 8566, cost 14.41 s 2023-09-05 14:21:28,153 44k INFO ====> Epoch: 8567, cost 14.31 s 2023-09-05 14:21:42,594 44k INFO ====> Epoch: 8568, cost 14.44 s 2023-09-05 14:21:57,076 44k INFO ====> Epoch: 8569, cost 14.48 s 2023-09-05 14:22:11,709 44k INFO ====> Epoch: 8570, cost 14.63 s 2023-09-05 14:22:26,099 44k INFO ====> Epoch: 8571, cost 14.39 s 2023-09-05 14:22:32,344 44k INFO Train Epoch: 8572 [38%] 2023-09-05 14:22:32,345 44k INFO Losses: [2.1320581436157227, 2.783221960067749, 9.590513229370117, 14.424905776977539, 0.6393248438835144], step: 180000, lr: 3.4251426705292365e-05, reference_loss: 29.570022583007812 2023-09-05 14:22:37,845 44k INFO Saving model and optimizer state at iteration 8572 to ./logs\44k\G_180000.pth 2023-09-05 14:22:38,476 44k INFO Saving model and optimizer state at iteration 8572 to ./logs\44k\D_180000.pth 2023-09-05 14:22:39,741 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_177600.pth 2023-09-05 14:22:39,781 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_177600.pth 2023-09-05 14:22:48,065 44k INFO ====> Epoch: 8572, cost 21.97 s 2023-09-05 14:23:02,816 44k INFO ====> Epoch: 8573, cost 14.75 s 2023-09-05 14:23:17,240 44k INFO ====> Epoch: 8574, cost 14.42 s 2023-09-05 14:23:31,726 44k INFO ====> Epoch: 8575, cost 14.49 s 2023-09-05 14:23:46,119 44k INFO ====> Epoch: 8576, cost 14.39 s 2023-09-05 14:24:00,993 44k INFO ====> Epoch: 8577, cost 14.87 s 2023-09-05 14:24:15,590 44k INFO ====> Epoch: 8578, cost 14.60 s 2023-09-05 14:24:29,999 44k INFO ====> Epoch: 8579, cost 14.41 s 2023-09-05 14:24:44,647 44k INFO ====> Epoch: 8580, cost 14.65 s 2023-09-05 14:24:58,735 44k INFO Train Epoch: 8581 [90%] 2023-09-05 14:24:58,736 44k INFO Losses: [2.034755229949951, 2.843478202819824, 11.784122467041016, 16.145179748535156, 0.7646368741989136], step: 180200, lr: 3.4212913111058096e-05, reference_loss: 33.57217025756836 2023-09-05 14:24:59,423 44k INFO ====> Epoch: 8581, cost 14.78 s 2023-09-05 14:25:14,165 44k INFO ====> Epoch: 8582, cost 14.74 s 2023-09-05 14:25:28,654 44k INFO ====> Epoch: 8583, cost 14.49 s 2023-09-05 14:25:43,265 44k INFO ====> Epoch: 8584, cost 14.61 s 2023-09-05 14:25:57,918 44k INFO ====> Epoch: 8585, cost 14.65 s 2023-09-05 14:26:12,527 44k INFO ====> Epoch: 8586, cost 14.61 s 2023-09-05 14:26:27,094 44k INFO ====> Epoch: 8587, cost 14.57 s 2023-09-05 14:26:41,651 44k INFO ====> Epoch: 8588, cost 14.56 s 2023-09-05 14:26:56,211 44k INFO ====> Epoch: 8589, cost 14.56 s 2023-09-05 14:27:10,875 44k INFO ====> Epoch: 8590, cost 14.66 s 2023-09-05 14:27:17,765 44k INFO Train Epoch: 8591 [43%] 2023-09-05 14:27:17,766 44k INFO Losses: [2.1969807147979736, 2.827565908432007, 9.387439727783203, 15.368794441223145, 0.5513136386871338], step: 180400, lr: 3.41701710176069e-05, reference_loss: 30.332094192504883 2023-09-05 14:27:25,375 44k INFO ====> Epoch: 8591, cost 14.50 s 2023-09-05 14:27:39,710 44k INFO ====> Epoch: 8592, cost 14.33 s 2023-09-05 14:27:54,177 44k INFO ====> Epoch: 8593, cost 14.47 s 2023-09-05 14:28:08,717 44k INFO ====> Epoch: 8594, cost 14.54 s 2023-09-05 14:28:23,174 44k INFO ====> Epoch: 8595, cost 14.46 s 2023-09-05 14:28:37,582 44k INFO ====> Epoch: 8596, cost 14.41 s 2023-09-05 14:28:52,096 44k INFO ====> Epoch: 8597, cost 14.51 s 2023-09-05 14:29:06,696 44k INFO ====> Epoch: 8598, cost 14.60 s 2023-09-05 14:29:21,215 44k INFO ====> Epoch: 8599, cost 14.52 s 2023-09-05 14:29:35,492 44k INFO Train Epoch: 8600 [95%] 2023-09-05 14:29:35,492 44k INFO Losses: [2.133768320083618, 2.81363844871521, 11.19051742553711, 14.634246826171875, 0.49449843168258667], step: 180600, lr: 3.413174879032828e-05, reference_loss: 31.26667022705078 2023-09-05 14:29:35,758 44k INFO ====> Epoch: 8600, cost 14.54 s 2023-09-05 14:29:50,529 44k INFO ====> Epoch: 8601, cost 14.77 s 2023-09-05 14:30:05,228 44k INFO ====> Epoch: 8602, cost 14.70 s 2023-09-05 14:30:19,712 44k INFO ====> Epoch: 8603, cost 14.48 s 2023-09-05 14:30:34,201 44k INFO ====> Epoch: 8604, cost 14.49 s 2023-09-05 14:30:48,770 44k INFO ====> Epoch: 8605, cost 14.57 s 2023-09-05 14:31:03,291 44k INFO ====> Epoch: 8606, cost 14.52 s 2023-09-05 14:31:17,745 44k INFO ====> Epoch: 8607, cost 14.45 s 2023-09-05 14:31:32,058 44k INFO ====> Epoch: 8608, cost 14.31 s 2023-09-05 14:31:46,583 44k INFO ====> Epoch: 8609, cost 14.52 s 2023-09-05 14:31:54,426 44k INFO Train Epoch: 8610 [48%] 2023-09-05 14:31:54,426 44k INFO Losses: [2.015627384185791, 2.933802843093872, 10.535993576049805, 16.312238693237305, 0.6556849479675293], step: 180800, lr: 3.408910809522834e-05, reference_loss: 32.453346252441406 2023-09-05 14:31:59,953 44k INFO Saving model and optimizer state at iteration 8610 to ./logs\44k\G_180800.pth 2023-09-05 14:32:00,582 44k INFO Saving model and optimizer state at iteration 8610 to ./logs\44k\D_180800.pth 2023-09-05 14:32:01,817 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_178400.pth 2023-09-05 14:32:01,853 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_178400.pth 2023-09-05 14:32:08,611 44k INFO ====> Epoch: 8610, cost 22.03 s 2023-09-05 14:32:22,930 44k INFO ====> Epoch: 8611, cost 14.32 s 2023-09-05 14:32:37,396 44k INFO ====> Epoch: 8612, cost 14.47 s 2023-09-05 14:32:51,974 44k INFO ====> Epoch: 8613, cost 14.58 s 2023-09-05 14:33:06,615 44k INFO ====> Epoch: 8614, cost 14.64 s 2023-09-05 14:33:21,138 44k INFO ====> Epoch: 8615, cost 14.52 s 2023-09-05 14:33:35,366 44k INFO ====> Epoch: 8616, cost 14.23 s 2023-09-05 14:33:49,772 44k INFO ====> Epoch: 8617, cost 14.41 s 2023-09-05 14:34:04,433 44k INFO ====> Epoch: 8618, cost 14.66 s 2023-09-05 14:34:18,830 44k INFO ====> Epoch: 8619, cost 14.40 s 2023-09-05 14:34:19,534 44k INFO Train Epoch: 8620 [0%] 2023-09-05 14:34:19,534 44k INFO Losses: [2.4706268310546875, 2.4761545658111572, 7.917270660400391, 13.686111450195312, 0.5854517221450806], step: 181000, lr: 3.4046520671025526e-05, reference_loss: 27.1356143951416 2023-09-05 14:34:33,608 44k INFO ====> Epoch: 8620, cost 14.78 s 2023-09-05 14:34:48,305 44k INFO ====> Epoch: 8621, cost 14.70 s 2023-09-05 14:35:03,030 44k INFO ====> Epoch: 8622, cost 14.73 s 2023-09-05 14:35:17,466 44k INFO ====> Epoch: 8623, cost 14.44 s 2023-09-05 14:35:31,861 44k INFO ====> Epoch: 8624, cost 14.40 s 2023-09-05 14:35:46,339 44k INFO ====> Epoch: 8625, cost 14.48 s 2023-09-05 14:36:00,838 44k INFO ====> Epoch: 8626, cost 14.50 s 2023-09-05 14:36:15,532 44k INFO ====> Epoch: 8627, cost 14.69 s 2023-09-05 14:36:29,973 44k INFO ====> Epoch: 8628, cost 14.44 s 2023-09-05 14:36:38,424 44k INFO Train Epoch: 8629 [52%] 2023-09-05 14:36:38,424 44k INFO Losses: [2.192863941192627, 2.8927953243255615, 8.572771072387695, 15.531146049499512, 0.7003327012062073], step: 181200, lr: 3.400823748085377e-05, reference_loss: 29.889907836914062 2023-09-05 14:36:44,812 44k INFO ====> Epoch: 8629, cost 14.84 s 2023-09-05 14:36:59,446 44k INFO ====> Epoch: 8630, cost 14.63 s 2023-09-05 14:37:13,970 44k INFO ====> Epoch: 8631, cost 14.52 s 2023-09-05 14:37:28,286 44k INFO ====> Epoch: 8632, cost 14.32 s 2023-09-05 14:37:42,707 44k INFO ====> Epoch: 8633, cost 14.42 s 2023-09-05 14:37:57,207 44k INFO ====> Epoch: 8634, cost 14.50 s 2023-09-05 14:38:11,750 44k INFO ====> Epoch: 8635, cost 14.54 s 2023-09-05 14:38:26,256 44k INFO ====> Epoch: 8636, cost 14.51 s 2023-09-05 14:38:40,803 44k INFO ====> Epoch: 8637, cost 14.55 s 2023-09-05 14:38:55,365 44k INFO ====> Epoch: 8638, cost 14.56 s 2023-09-05 14:38:56,795 44k INFO Train Epoch: 8639 [5%] 2023-09-05 14:38:56,795 44k INFO Losses: [1.9944120645523071, 3.1305105686187744, 10.67508316040039, 16.123886108398438, 0.5193041563034058], step: 181400, lr: 3.3965751088075745e-05, reference_loss: 32.44319534301758 2023-09-05 14:39:10,340 44k INFO ====> Epoch: 8639, cost 14.97 s 2023-09-05 14:39:24,731 44k INFO ====> Epoch: 8640, cost 14.39 s 2023-09-05 14:39:39,124 44k INFO ====> Epoch: 8641, cost 14.39 s 2023-09-05 14:39:53,546 44k INFO ====> Epoch: 8642, cost 14.42 s 2023-09-05 14:40:08,165 44k INFO ====> Epoch: 8643, cost 14.62 s 2023-09-05 14:40:22,585 44k INFO ====> Epoch: 8644, cost 14.42 s 2023-09-05 14:40:37,038 44k INFO ====> Epoch: 8645, cost 14.45 s 2023-09-05 14:40:51,888 44k INFO ====> Epoch: 8646, cost 14.85 s 2023-09-05 14:41:06,674 44k INFO ====> Epoch: 8647, cost 14.79 s 2023-09-05 14:41:15,858 44k INFO Train Epoch: 8648 [57%] 2023-09-05 14:41:15,858 44k INFO Losses: [2.190922975540161, 2.802445888519287, 8.045859336853027, 14.32558536529541, 0.767722487449646], step: 181600, lr: 3.392755871826517e-05, reference_loss: 28.132537841796875 2023-09-05 14:41:21,314 44k INFO Saving model and optimizer state at iteration 8648 to ./logs\44k\G_181600.pth 2023-09-05 14:41:21,941 44k INFO Saving model and optimizer state at iteration 8648 to ./logs\44k\D_181600.pth 2023-09-05 14:41:23,613 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_179200.pth 2023-09-05 14:41:23,650 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_179200.pth 2023-09-05 14:41:29,043 44k INFO ====> Epoch: 8648, cost 22.37 s 2023-09-05 14:41:43,453 44k INFO ====> Epoch: 8649, cost 14.41 s 2023-09-05 14:41:58,162 44k INFO ====> Epoch: 8650, cost 14.71 s 2023-09-05 14:42:12,963 44k INFO ====> Epoch: 8651, cost 14.80 s 2023-09-05 14:42:27,480 44k INFO ====> Epoch: 8652, cost 14.52 s 2023-09-05 14:42:41,968 44k INFO ====> Epoch: 8653, cost 14.49 s 2023-09-05 14:42:56,376 44k INFO ====> Epoch: 8654, cost 14.41 s 2023-09-05 14:43:10,847 44k INFO ====> Epoch: 8655, cost 14.47 s 2023-09-05 14:43:25,244 44k INFO ====> Epoch: 8656, cost 14.40 s 2023-09-05 14:43:39,734 44k INFO ====> Epoch: 8657, cost 14.49 s 2023-09-05 14:43:41,824 44k INFO Train Epoch: 8658 [10%] 2023-09-05 14:43:41,824 44k INFO Losses: [2.2014219760894775, 2.7909417152404785, 9.700486183166504, 15.007567405700684, 0.6827915906906128], step: 181800, lr: 3.3885173117232014e-05, reference_loss: 30.383207321166992 2023-09-05 14:43:54,663 44k INFO ====> Epoch: 8658, cost 14.93 s 2023-09-05 14:44:09,492 44k INFO ====> Epoch: 8659, cost 14.83 s 2023-09-05 14:44:23,863 44k INFO ====> Epoch: 8660, cost 14.37 s 2023-09-05 14:44:38,328 44k INFO ====> Epoch: 8661, cost 14.46 s 2023-09-05 14:44:52,743 44k INFO ====> Epoch: 8662, cost 14.42 s 2023-09-05 14:45:07,173 44k INFO ====> Epoch: 8663, cost 14.43 s 2023-09-05 14:45:21,559 44k INFO ====> Epoch: 8664, cost 14.39 s 2023-09-05 14:45:36,008 44k INFO ====> Epoch: 8665, cost 14.45 s 2023-09-05 14:45:50,543 44k INFO ====> Epoch: 8666, cost 14.53 s 2023-09-05 14:46:00,489 44k INFO Train Epoch: 8667 [62%] 2023-09-05 14:46:00,489 44k INFO Losses: [2.0122015476226807, 2.8270790576934814, 12.159845352172852, 16.97227668762207, 0.5519348382949829], step: 182000, lr: 3.384707135232675e-05, reference_loss: 34.523338317871094 2023-09-05 14:46:05,477 44k INFO ====> Epoch: 8667, cost 14.93 s 2023-09-05 14:46:19,952 44k INFO ====> Epoch: 8668, cost 14.48 s 2023-09-05 14:46:34,327 44k INFO ====> Epoch: 8669, cost 14.37 s 2023-09-05 14:46:48,968 44k INFO ====> Epoch: 8670, cost 14.64 s 2023-09-05 14:47:03,838 44k INFO ====> Epoch: 8671, cost 14.87 s 2023-09-05 14:47:18,370 44k INFO ====> Epoch: 8672, cost 14.53 s 2023-09-05 14:47:32,804 44k INFO ====> Epoch: 8673, cost 14.43 s 2023-09-05 14:47:47,511 44k INFO ====> Epoch: 8674, cost 14.71 s 2023-09-05 14:48:02,271 44k INFO ====> Epoch: 8675, cost 14.76 s 2023-09-05 14:48:16,983 44k INFO ====> Epoch: 8676, cost 14.71 s 2023-09-05 14:48:19,802 44k INFO Train Epoch: 8677 [14%] 2023-09-05 14:48:19,803 44k INFO Losses: [1.9176785945892334, 3.026883840560913, 11.233968734741211, 14.90006160736084, 0.5819399356842041], step: 182200, lr: 3.380478630392721e-05, reference_loss: 31.660533905029297 2023-09-05 14:48:31,820 44k INFO ====> Epoch: 8677, cost 14.84 s 2023-09-05 14:48:46,434 44k INFO ====> Epoch: 8678, cost 14.61 s 2023-09-05 14:49:00,998 44k INFO ====> Epoch: 8679, cost 14.56 s 2023-09-05 14:49:15,639 44k INFO ====> Epoch: 8680, cost 14.64 s 2023-09-05 14:49:29,942 44k INFO ====> Epoch: 8681, cost 14.30 s 2023-09-05 14:49:44,430 44k INFO ====> Epoch: 8682, cost 14.49 s 2023-09-05 14:49:58,973 44k INFO ====> Epoch: 8683, cost 14.54 s 2023-09-05 14:50:13,491 44k INFO ====> Epoch: 8684, cost 14.52 s 2023-09-05 14:50:27,985 44k INFO ====> Epoch: 8685, cost 14.49 s 2023-09-05 14:50:38,444 44k INFO Train Epoch: 8686 [67%] 2023-09-05 14:50:38,445 44k INFO Losses: [2.254667282104492, 2.708862781524658, 9.194823265075684, 16.432300567626953, 0.609559953212738], step: 182400, lr: 3.376677492898253e-05, reference_loss: 31.200214385986328 2023-09-05 14:50:43,981 44k INFO Saving model and optimizer state at iteration 8686 to ./logs\44k\G_182400.pth 2023-09-05 14:50:44,609 44k INFO Saving model and optimizer state at iteration 8686 to ./logs\44k\D_182400.pth 2023-09-05 14:50:46,221 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_180000.pth 2023-09-05 14:50:46,256 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_180000.pth 2023-09-05 14:50:50,246 44k INFO ====> Epoch: 8686, cost 22.26 s 2023-09-05 14:51:04,758 44k INFO ====> Epoch: 8687, cost 14.51 s 2023-09-05 14:51:19,209 44k INFO ====> Epoch: 8688, cost 14.45 s 2023-09-05 14:51:33,463 44k INFO ====> Epoch: 8689, cost 14.25 s 2023-09-05 14:51:47,895 44k INFO ====> Epoch: 8690, cost 14.43 s 2023-09-05 14:52:02,448 44k INFO ====> Epoch: 8691, cost 14.55 s 2023-09-05 14:52:16,865 44k INFO ====> Epoch: 8692, cost 14.42 s 2023-09-05 14:52:31,437 44k INFO ====> Epoch: 8693, cost 14.57 s 2023-09-05 14:52:46,000 44k INFO ====> Epoch: 8694, cost 14.56 s 2023-09-05 14:53:00,694 44k INFO ====> Epoch: 8695, cost 14.69 s 2023-09-05 14:53:04,239 44k INFO Train Epoch: 8696 [19%] 2023-09-05 14:53:04,239 44k INFO Losses: [2.3284311294555664, 2.595777988433838, 8.388832092285156, 15.883060455322266, 0.6313416957855225], step: 182600, lr: 3.372459019467255e-05, reference_loss: 29.827444076538086 2023-09-05 14:53:15,403 44k INFO ====> Epoch: 8696, cost 14.71 s 2023-09-05 14:53:29,769 44k INFO ====> Epoch: 8697, cost 14.37 s 2023-09-05 14:53:44,236 44k INFO ====> Epoch: 8698, cost 14.47 s 2023-09-05 14:53:58,925 44k INFO ====> Epoch: 8699, cost 14.69 s 2023-09-05 14:54:13,534 44k INFO ====> Epoch: 8700, cost 14.61 s 2023-09-05 14:54:28,081 44k INFO ====> Epoch: 8701, cost 14.55 s 2023-09-05 14:54:42,505 44k INFO ====> Epoch: 8702, cost 14.42 s 2023-09-05 14:54:56,991 44k INFO ====> Epoch: 8703, cost 14.49 s 2023-09-05 14:55:11,503 44k INFO ====> Epoch: 8704, cost 14.51 s 2023-09-05 14:55:22,640 44k INFO Train Epoch: 8705 [71%] 2023-09-05 14:55:22,640 44k INFO Losses: [1.9399609565734863, 2.9773740768432617, 10.676583290100098, 14.30759334564209, 0.577724039554596], step: 182800, lr: 3.368666899525361e-05, reference_loss: 30.479236602783203 2023-09-05 14:55:26,103 44k INFO ====> Epoch: 8705, cost 14.60 s 2023-09-05 14:55:40,410 44k INFO ====> Epoch: 8706, cost 14.31 s 2023-09-05 14:55:55,250 44k INFO ====> Epoch: 8707, cost 14.84 s 2023-09-05 14:56:09,959 44k INFO ====> Epoch: 8708, cost 14.71 s 2023-09-05 14:56:24,502 44k INFO ====> Epoch: 8709, cost 14.54 s 2023-09-05 14:56:39,040 44k INFO ====> Epoch: 8710, cost 14.54 s 2023-09-05 14:56:53,588 44k INFO ====> Epoch: 8711, cost 14.55 s 2023-09-05 14:57:08,127 44k INFO ====> Epoch: 8712, cost 14.54 s 2023-09-05 14:57:22,560 44k INFO ====> Epoch: 8713, cost 14.43 s 2023-09-05 14:57:36,860 44k INFO ====> Epoch: 8714, cost 14.30 s 2023-09-05 14:57:41,165 44k INFO Train Epoch: 8715 [24%] 2023-09-05 14:57:41,165 44k INFO Losses: [2.305997133255005, 2.355867624282837, 7.887928485870361, 13.794655799865723, 0.5729242563247681], step: 183000, lr: 3.3644584337055094e-05, reference_loss: 26.91737174987793 2023-09-05 14:57:51,881 44k INFO ====> Epoch: 8715, cost 15.02 s 2023-09-05 14:58:06,497 44k INFO ====> Epoch: 8716, cost 14.62 s 2023-09-05 14:58:21,440 44k INFO ====> Epoch: 8717, cost 14.94 s 2023-09-05 14:58:35,735 44k INFO ====> Epoch: 8718, cost 14.29 s 2023-09-05 14:58:50,314 44k INFO ====> Epoch: 8719, cost 14.58 s 2023-09-05 14:59:04,941 44k INFO ====> Epoch: 8720, cost 14.63 s 2023-09-05 14:59:19,459 44k INFO ====> Epoch: 8721, cost 14.52 s 2023-09-05 14:59:33,786 44k INFO ====> Epoch: 8722, cost 14.33 s 2023-09-05 14:59:48,428 44k INFO ====> Epoch: 8723, cost 14.64 s 2023-09-05 15:00:00,536 44k INFO Train Epoch: 8724 [76%] 2023-09-05 15:00:00,536 44k INFO Losses: [1.848215937614441, 2.9469401836395264, 12.392822265625, 17.399524688720703, 0.5962100625038147], step: 183200, lr: 3.360675309923581e-05, reference_loss: 35.183712005615234 2023-09-05 15:00:06,125 44k INFO Saving model and optimizer state at iteration 8724 to ./logs\44k\G_183200.pth 2023-09-05 15:00:06,735 44k INFO Saving model and optimizer state at iteration 8724 to ./logs\44k\D_183200.pth 2023-09-05 15:00:08,000 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_180800.pth 2023-09-05 15:00:08,035 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_180800.pth 2023-09-05 15:00:10,610 44k INFO ====> Epoch: 8724, cost 22.18 s 2023-09-05 15:00:25,096 44k INFO ====> Epoch: 8725, cost 14.49 s 2023-09-05 15:00:39,677 44k INFO ====> Epoch: 8726, cost 14.58 s 2023-09-05 15:00:54,355 44k INFO ====> Epoch: 8727, cost 14.68 s 2023-09-05 15:01:09,022 44k INFO ====> Epoch: 8728, cost 14.67 s 2023-09-05 15:01:23,350 44k INFO ====> Epoch: 8729, cost 14.33 s 2023-09-05 15:01:37,803 44k INFO ====> Epoch: 8730, cost 14.45 s 2023-09-05 15:01:52,470 44k INFO ====> Epoch: 8731, cost 14.67 s 2023-09-05 15:02:07,143 44k INFO ====> Epoch: 8732, cost 14.67 s 2023-09-05 15:02:21,569 44k INFO ====> Epoch: 8733, cost 14.43 s 2023-09-05 15:02:26,477 44k INFO Train Epoch: 8734 [29%] 2023-09-05 15:02:26,477 44k INFO Losses: [2.227611780166626, 2.599717617034912, 7.487209320068359, 12.92676830291748, 0.6003522276878357], step: 183400, lr: 3.356476827973518e-05, reference_loss: 25.841657638549805 2023-09-05 15:02:36,311 44k INFO ====> Epoch: 8734, cost 14.74 s 2023-09-05 15:02:50,947 44k INFO ====> Epoch: 8735, cost 14.64 s 2023-09-05 15:03:05,480 44k INFO ====> Epoch: 8736, cost 14.53 s 2023-09-05 15:03:19,977 44k INFO ====> Epoch: 8737, cost 14.50 s 2023-09-05 15:03:34,435 44k INFO ====> Epoch: 8738, cost 14.46 s 2023-09-05 15:03:48,769 44k INFO ====> Epoch: 8739, cost 14.33 s 2023-09-05 15:04:03,348 44k INFO ====> Epoch: 8740, cost 14.58 s 2023-09-05 15:04:17,972 44k INFO ====> Epoch: 8741, cost 14.62 s 2023-09-05 15:04:32,571 44k INFO ====> Epoch: 8742, cost 14.60 s 2023-09-05 15:04:45,322 44k INFO Train Epoch: 8743 [81%] 2023-09-05 15:04:45,322 44k INFO Losses: [2.2094080448150635, 2.4833555221557617, 11.380241394042969, 15.302759170532227, 0.6340608596801758], step: 183600, lr: 3.352702679009692e-05, reference_loss: 32.00982666015625 2023-09-05 15:04:47,527 44k INFO ====> Epoch: 8743, cost 14.96 s 2023-09-05 15:05:02,321 44k INFO ====> Epoch: 8744, cost 14.79 s 2023-09-05 15:05:16,741 44k INFO ====> Epoch: 8745, cost 14.42 s 2023-09-05 15:05:31,069 44k INFO ====> Epoch: 8746, cost 14.33 s 2023-09-05 15:05:45,709 44k INFO ====> Epoch: 8747, cost 14.64 s 2023-09-05 15:06:00,530 44k INFO ====> Epoch: 8748, cost 14.82 s 2023-09-05 15:06:15,288 44k INFO ====> Epoch: 8749, cost 14.76 s 2023-09-05 15:06:29,822 44k INFO ====> Epoch: 8750, cost 14.53 s 2023-09-05 15:06:44,312 44k INFO ====> Epoch: 8751, cost 14.49 s 2023-09-05 15:06:58,948 44k INFO ====> Epoch: 8752, cost 14.64 s 2023-09-05 15:07:04,617 44k INFO Train Epoch: 8753 [33%] 2023-09-05 15:07:04,617 44k INFO Losses: [2.2744948863983154, 2.330646276473999, 9.28000545501709, 14.949790954589844, 0.5584642887115479], step: 183800, lr: 3.348514157244382e-05, reference_loss: 29.393402099609375 2023-09-05 15:07:13,903 44k INFO ====> Epoch: 8753, cost 14.96 s 2023-09-05 15:07:28,344 44k INFO ====> Epoch: 8754, cost 14.44 s 2023-09-05 15:07:42,827 44k INFO ====> Epoch: 8755, cost 14.48 s 2023-09-05 15:07:57,463 44k INFO ====> Epoch: 8756, cost 14.64 s 2023-09-05 15:08:12,198 44k INFO ====> Epoch: 8757, cost 14.74 s 2023-09-05 15:08:26,809 44k INFO ====> Epoch: 8758, cost 14.61 s 2023-09-05 15:08:41,368 44k INFO ====> Epoch: 8759, cost 14.56 s 2023-09-05 15:08:55,898 44k INFO ====> Epoch: 8760, cost 14.53 s 2023-09-05 15:09:10,616 44k INFO ====> Epoch: 8761, cost 14.72 s 2023-09-05 15:09:23,725 44k INFO Train Epoch: 8762 [86%] 2023-09-05 15:09:23,726 44k INFO Losses: [2.000810146331787, 2.973094940185547, 11.838509559631348, 15.662287712097168, 0.5912960171699524], step: 184000, lr: 3.344748961807432e-05, reference_loss: 33.06599807739258 2023-09-05 15:09:29,230 44k INFO Saving model and optimizer state at iteration 8762 to ./logs\44k\G_184000.pth 2023-09-05 15:09:29,894 44k INFO Saving model and optimizer state at iteration 8762 to ./logs\44k\D_184000.pth 2023-09-05 15:09:31,017 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_181600.pth 2023-09-05 15:09:31,056 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_181600.pth 2023-09-05 15:09:32,218 44k INFO ====> Epoch: 8762, cost 21.60 s 2023-09-05 15:09:46,843 44k INFO ====> Epoch: 8763, cost 14.63 s 2023-09-05 15:10:01,390 44k INFO ====> Epoch: 8764, cost 14.55 s 2023-09-05 15:10:16,096 44k INFO ====> Epoch: 8765, cost 14.71 s 2023-09-05 15:10:30,587 44k INFO ====> Epoch: 8766, cost 14.49 s 2023-09-05 15:10:44,879 44k INFO ====> Epoch: 8767, cost 14.29 s 2023-09-05 15:10:59,380 44k INFO ====> Epoch: 8768, cost 14.50 s 2023-09-05 15:11:13,981 44k INFO ====> Epoch: 8769, cost 14.60 s 2023-09-05 15:11:28,323 44k INFO ====> Epoch: 8770, cost 14.34 s 2023-09-05 15:11:42,705 44k INFO ====> Epoch: 8771, cost 14.38 s 2023-09-05 15:11:49,153 44k INFO Train Epoch: 8772 [38%] 2023-09-05 15:11:49,153 44k INFO Losses: [2.262303113937378, 2.468768835067749, 9.316252708435059, 15.524975776672363, 0.5902293920516968], step: 184200, lr: 3.340570376598031e-05, reference_loss: 30.16252899169922 2023-09-05 15:11:57,811 44k INFO ====> Epoch: 8772, cost 15.11 s 2023-09-05 15:12:12,588 44k INFO ====> Epoch: 8773, cost 14.78 s 2023-09-05 15:12:27,176 44k INFO ====> Epoch: 8774, cost 14.59 s 2023-09-05 15:12:41,665 44k INFO ====> Epoch: 8775, cost 14.49 s 2023-09-05 15:12:56,294 44k INFO ====> Epoch: 8776, cost 14.63 s 2023-09-05 15:13:10,990 44k INFO ====> Epoch: 8777, cost 14.70 s 2023-09-05 15:13:25,189 44k INFO ====> Epoch: 8778, cost 14.20 s 2023-09-05 15:13:39,556 44k INFO ====> Epoch: 8779, cost 14.37 s 2023-09-05 15:13:54,352 44k INFO ====> Epoch: 8780, cost 14.80 s 2023-09-05 15:14:08,763 44k INFO Train Epoch: 8781 [90%] 2023-09-05 15:14:08,764 44k INFO Losses: [2.2131404876708984, 2.5874509811401367, 9.237506866455078, 15.71716022491455, 0.5030993223190308], step: 184400, lr: 3.336814113447235e-05, reference_loss: 30.258358001708984 2023-09-05 15:14:09,466 44k INFO ====> Epoch: 8781, cost 15.11 s 2023-09-05 15:14:23,892 44k INFO ====> Epoch: 8782, cost 14.43 s 2023-09-05 15:14:38,442 44k INFO ====> Epoch: 8783, cost 14.55 s 2023-09-05 15:14:52,883 44k INFO ====> Epoch: 8784, cost 14.44 s 2023-09-05 15:15:07,429 44k INFO ====> Epoch: 8785, cost 14.55 s 2023-09-05 15:15:21,849 44k INFO ====> Epoch: 8786, cost 14.42 s 2023-09-05 15:15:36,155 44k INFO ====> Epoch: 8787, cost 14.31 s 2023-09-05 15:15:50,893 44k INFO ====> Epoch: 8788, cost 14.74 s 2023-09-05 15:16:05,518 44k INFO ====> Epoch: 8789, cost 14.63 s 2023-09-05 15:16:20,063 44k INFO ====> Epoch: 8790, cost 14.54 s 2023-09-05 15:16:27,139 44k INFO Train Epoch: 8791 [43%] 2023-09-05 15:16:27,139 44k INFO Losses: [2.1481924057006836, 2.63897967338562, 8.937910079956055, 13.988099098205566, 0.5150738954544067], step: 184600, lr: 3.332645441220954e-05, reference_loss: 28.228254318237305 2023-09-05 15:16:34,743 44k INFO ====> Epoch: 8791, cost 14.68 s 2023-09-05 15:16:49,386 44k INFO ====> Epoch: 8792, cost 14.64 s 2023-09-05 15:17:04,070 44k INFO ====> Epoch: 8793, cost 14.68 s 2023-09-05 15:17:18,732 44k INFO ====> Epoch: 8794, cost 14.66 s 2023-09-05 15:17:33,086 44k INFO ====> Epoch: 8795, cost 14.35 s 2023-09-05 15:17:47,724 44k INFO ====> Epoch: 8796, cost 14.64 s 2023-09-05 15:18:02,617 44k INFO ====> Epoch: 8797, cost 14.89 s 2023-09-05 15:18:17,284 44k INFO ====> Epoch: 8798, cost 14.67 s 2023-09-05 15:18:31,710 44k INFO ====> Epoch: 8799, cost 14.43 s 2023-09-05 15:18:46,296 44k INFO Train Epoch: 8800 [95%] 2023-09-05 15:18:46,296 44k INFO Losses: [2.170577049255371, 2.624476432800293, 11.2333402633667, 15.452105522155762, 0.3033771514892578], step: 184800, lr: 3.328898089165982e-05, reference_loss: 31.783876419067383 2023-09-05 15:18:51,792 44k INFO Saving model and optimizer state at iteration 8800 to ./logs\44k\G_184800.pth 2023-09-05 15:18:52,476 44k INFO Saving model and optimizer state at iteration 8800 to ./logs\44k\D_184800.pth 2023-09-05 15:18:53,645 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_182400.pth 2023-09-05 15:18:53,682 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_182400.pth 2023-09-05 15:18:53,682 44k INFO ====> Epoch: 8800, cost 21.97 s 2023-09-05 15:19:08,421 44k INFO ====> Epoch: 8801, cost 14.74 s 2023-09-05 15:19:22,875 44k INFO ====> Epoch: 8802, cost 14.45 s 2023-09-05 15:19:37,352 44k INFO ====> Epoch: 8803, cost 14.48 s 2023-09-05 15:19:52,033 44k INFO ====> Epoch: 8804, cost 14.68 s 2023-09-05 15:20:06,641 44k INFO ====> Epoch: 8805, cost 14.61 s 2023-09-05 15:20:21,115 44k INFO ====> Epoch: 8806, cost 14.47 s 2023-09-05 15:20:35,670 44k INFO ====> Epoch: 8807, cost 14.56 s 2023-09-05 15:20:50,065 44k INFO ====> Epoch: 8808, cost 14.39 s 2023-09-05 15:21:04,714 44k INFO ====> Epoch: 8809, cost 14.65 s 2023-09-05 15:21:12,488 44k INFO Train Epoch: 8810 [48%] 2023-09-05 15:21:12,489 44k INFO Losses: [1.9657466411590576, 3.039363384246826, 12.638136863708496, 16.36385726928711, 0.7464240193367004], step: 185000, lr: 3.3247393064059533e-05, reference_loss: 34.7535285949707 2023-09-05 15:21:19,380 44k INFO ====> Epoch: 8810, cost 14.67 s 2023-09-05 15:21:33,952 44k INFO ====> Epoch: 8811, cost 14.57 s 2023-09-05 15:21:48,506 44k INFO ====> Epoch: 8812, cost 14.55 s 2023-09-05 15:22:03,022 44k INFO ====> Epoch: 8813, cost 14.52 s 2023-09-05 15:22:17,574 44k INFO ====> Epoch: 8814, cost 14.55 s 2023-09-05 15:22:32,165 44k INFO ====> Epoch: 8815, cost 14.59 s 2023-09-05 15:22:46,604 44k INFO ====> Epoch: 8816, cost 14.44 s 2023-09-05 15:23:01,238 44k INFO ====> Epoch: 8817, cost 14.63 s 2023-09-05 15:23:15,761 44k INFO ====> Epoch: 8818, cost 14.52 s 2023-09-05 15:23:30,215 44k INFO ====> Epoch: 8819, cost 14.45 s 2023-09-05 15:23:30,912 44k INFO Train Epoch: 8820 [0%] 2023-09-05 15:23:30,912 44k INFO Losses: [2.371316909790039, 2.722346544265747, 8.606361389160156, 13.719598770141602, 0.6095831394195557], step: 185200, lr: 3.3205857192012045e-05, reference_loss: 28.029207229614258 2023-09-05 15:23:44,910 44k INFO ====> Epoch: 8820, cost 14.69 s 2023-09-05 15:23:59,683 44k INFO ====> Epoch: 8821, cost 14.77 s 2023-09-05 15:24:14,420 44k INFO ====> Epoch: 8822, cost 14.74 s 2023-09-05 15:24:28,818 44k INFO ====> Epoch: 8823, cost 14.40 s 2023-09-05 15:24:43,425 44k INFO ====> Epoch: 8824, cost 14.61 s 2023-09-05 15:24:58,121 44k INFO ====> Epoch: 8825, cost 14.70 s 2023-09-05 15:25:12,602 44k INFO ====> Epoch: 8826, cost 14.48 s 2023-09-05 15:25:27,007 44k INFO ====> Epoch: 8827, cost 14.41 s 2023-09-05 15:25:41,415 44k INFO ====> Epoch: 8828, cost 14.41 s 2023-09-05 15:25:49,911 44k INFO Train Epoch: 8829 [52%] 2023-09-05 15:25:49,911 44k INFO Losses: [2.1792824268341064, 2.8100428581237793, 9.607842445373535, 16.201257705688477, 0.48196107149124146], step: 185400, lr: 3.316851927551888e-05, reference_loss: 31.280385971069336 2023-09-05 15:25:56,283 44k INFO ====> Epoch: 8829, cost 14.87 s 2023-09-05 15:26:10,940 44k INFO ====> Epoch: 8830, cost 14.66 s 2023-09-05 15:26:25,629 44k INFO ====> Epoch: 8831, cost 14.69 s 2023-09-05 15:26:40,045 44k INFO ====> Epoch: 8832, cost 14.42 s 2023-09-05 15:26:54,589 44k INFO ====> Epoch: 8833, cost 14.54 s 2023-09-05 15:27:09,188 44k INFO ====> Epoch: 8834, cost 14.60 s 2023-09-05 15:27:23,658 44k INFO ====> Epoch: 8835, cost 14.47 s 2023-09-05 15:27:38,184 44k INFO ====> Epoch: 8836, cost 14.53 s 2023-09-05 15:27:52,770 44k INFO ====> Epoch: 8837, cost 14.59 s 2023-09-05 15:28:07,180 44k INFO ====> Epoch: 8838, cost 14.41 s 2023-09-05 15:28:08,636 44k INFO Train Epoch: 8839 [5%] 2023-09-05 15:28:08,636 44k INFO Losses: [2.159308671951294, 2.96433162689209, 10.315727233886719, 14.662374496459961, 0.6072954535484314], step: 185600, lr: 3.312708194026741e-05, reference_loss: 30.70903778076172 2023-09-05 15:28:14,134 44k INFO Saving model and optimizer state at iteration 8839 to ./logs\44k\G_185600.pth 2023-09-05 15:28:14,773 44k INFO Saving model and optimizer state at iteration 8839 to ./logs\44k\D_185600.pth 2023-09-05 15:28:16,013 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_183200.pth 2023-09-05 15:28:16,050 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_183200.pth 2023-09-05 15:28:29,164 44k INFO ====> Epoch: 8839, cost 21.98 s 2023-09-05 15:28:43,624 44k INFO ====> Epoch: 8840, cost 14.46 s 2023-09-05 15:28:58,355 44k INFO ====> Epoch: 8841, cost 14.73 s 2023-09-05 15:29:12,906 44k INFO ====> Epoch: 8842, cost 14.55 s 2023-09-05 15:29:27,405 44k INFO ====> Epoch: 8843, cost 14.50 s 2023-09-05 15:29:41,877 44k INFO ====> Epoch: 8844, cost 14.47 s 2023-09-05 15:29:56,853 44k INFO ====> Epoch: 8845, cost 14.98 s 2023-09-05 15:30:11,650 44k INFO ====> Epoch: 8846, cost 14.80 s 2023-09-05 15:30:26,091 44k INFO ====> Epoch: 8847, cost 14.44 s 2023-09-05 15:30:35,296 44k INFO Train Epoch: 8848 [57%] 2023-09-05 15:30:35,297 44k INFO Losses: [1.8348941802978516, 3.588078737258911, 10.599145889282227, 13.934420585632324, 0.5773572325706482], step: 185800, lr: 3.3089832601634286e-05, reference_loss: 30.53389549255371 2023-09-05 15:30:40,880 44k INFO ====> Epoch: 8848, cost 14.79 s 2023-09-05 15:30:55,450 44k INFO ====> Epoch: 8849, cost 14.57 s 2023-09-05 15:31:10,070 44k INFO ====> Epoch: 8850, cost 14.62 s 2023-09-05 15:31:24,583 44k INFO ====> Epoch: 8851, cost 14.51 s 2023-09-05 15:31:38,960 44k INFO ====> Epoch: 8852, cost 14.38 s 2023-09-05 15:31:53,713 44k INFO ====> Epoch: 8853, cost 14.75 s 2023-09-05 15:32:08,248 44k INFO ====> Epoch: 8854, cost 14.54 s 2023-09-05 15:32:22,804 44k INFO ====> Epoch: 8855, cost 14.56 s 2023-09-05 15:32:37,421 44k INFO ====> Epoch: 8856, cost 14.62 s 2023-09-05 15:32:51,913 44k INFO ====> Epoch: 8857, cost 14.49 s 2023-09-05 15:32:54,080 44k INFO Train Epoch: 8858 [10%] 2023-09-05 15:32:54,081 44k INFO Losses: [2.4313480854034424, 2.56097412109375, 7.0556511878967285, 13.672468185424805, 0.5165521502494812], step: 186000, lr: 3.304849356941704e-05, reference_loss: 26.23699378967285 2023-09-05 15:33:06,796 44k INFO ====> Epoch: 8858, cost 14.88 s 2023-09-05 15:33:21,088 44k INFO ====> Epoch: 8859, cost 14.29 s 2023-09-05 15:33:35,488 44k INFO ====> Epoch: 8860, cost 14.40 s 2023-09-05 15:33:50,104 44k INFO ====> Epoch: 8861, cost 14.62 s 2023-09-05 15:34:04,618 44k INFO ====> Epoch: 8862, cost 14.51 s 2023-09-05 15:34:19,168 44k INFO ====> Epoch: 8863, cost 14.55 s 2023-09-05 15:34:33,753 44k INFO ====> Epoch: 8864, cost 14.58 s 2023-09-05 15:34:48,140 44k INFO ====> Epoch: 8865, cost 14.39 s 2023-09-05 15:35:02,951 44k INFO ====> Epoch: 8866, cost 14.81 s 2023-09-05 15:35:12,864 44k INFO Train Epoch: 8867 [62%] 2023-09-05 15:35:12,864 44k INFO Losses: [2.0490596294403076, 2.868896007537842, 9.655680656433105, 14.747244834899902, 0.5142178535461426], step: 186200, lr: 3.301133259850806e-05, reference_loss: 29.835098266601562 2023-09-05 15:35:17,719 44k INFO ====> Epoch: 8867, cost 14.77 s 2023-09-05 15:35:32,084 44k INFO ====> Epoch: 8868, cost 14.37 s 2023-09-05 15:35:46,661 44k INFO ====> Epoch: 8869, cost 14.58 s 2023-09-05 15:36:01,322 44k INFO ====> Epoch: 8870, cost 14.66 s 2023-09-05 15:36:16,078 44k INFO ====> Epoch: 8871, cost 14.76 s 2023-09-05 15:36:30,443 44k INFO ====> Epoch: 8872, cost 14.36 s 2023-09-05 15:36:44,947 44k INFO ====> Epoch: 8873, cost 14.50 s 2023-09-05 15:36:59,443 44k INFO ====> Epoch: 8874, cost 14.50 s 2023-09-05 15:37:13,847 44k INFO ====> Epoch: 8875, cost 14.40 s 2023-09-05 15:37:28,110 44k INFO ====> Epoch: 8876, cost 14.26 s 2023-09-05 15:37:30,898 44k INFO Train Epoch: 8877 [14%] 2023-09-05 15:37:30,898 44k INFO Losses: [2.350405216217041, 2.7076165676116943, 10.344379425048828, 15.514447212219238, 0.6644659638404846], step: 186400, lr: 3.297009163611781e-05, reference_loss: 31.581314086914062 2023-09-05 15:37:36,372 44k INFO Saving model and optimizer state at iteration 8877 to ./logs\44k\G_186400.pth 2023-09-05 15:37:36,964 44k INFO Saving model and optimizer state at iteration 8877 to ./logs\44k\D_186400.pth 2023-09-05 15:37:38,097 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_184000.pth 2023-09-05 15:37:38,131 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_184000.pth 2023-09-05 15:37:50,060 44k INFO ====> Epoch: 8877, cost 21.95 s 2023-09-05 15:38:04,938 44k INFO ====> Epoch: 8878, cost 14.88 s 2023-09-05 15:38:19,535 44k INFO ====> Epoch: 8879, cost 14.60 s 2023-09-05 15:38:34,018 44k INFO ====> Epoch: 8880, cost 14.48 s 2023-09-05 15:38:48,672 44k INFO ====> Epoch: 8881, cost 14.65 s 2023-09-05 15:39:03,360 44k INFO ====> Epoch: 8882, cost 14.69 s 2023-09-05 15:39:17,997 44k INFO ====> Epoch: 8883, cost 14.64 s 2023-09-05 15:39:32,604 44k INFO ====> Epoch: 8884, cost 14.61 s 2023-09-05 15:39:47,192 44k INFO ====> Epoch: 8885, cost 14.59 s 2023-09-05 15:39:57,798 44k INFO Train Epoch: 8886 [67%] 2023-09-05 15:39:57,799 44k INFO Losses: [2.143954038619995, 2.7138984203338623, 9.87474250793457, 15.123591423034668, 0.5050166845321655], step: 186600, lr: 3.2933018823295575e-05, reference_loss: 30.361202239990234 2023-09-05 15:40:02,139 44k INFO ====> Epoch: 8886, cost 14.95 s 2023-09-05 15:40:16,720 44k INFO ====> Epoch: 8887, cost 14.58 s 2023-09-05 15:40:31,100 44k INFO ====> Epoch: 8888, cost 14.38 s 2023-09-05 15:40:45,688 44k INFO ====> Epoch: 8889, cost 14.59 s 2023-09-05 15:41:00,387 44k INFO ====> Epoch: 8890, cost 14.70 s 2023-09-05 15:41:14,911 44k INFO ====> Epoch: 8891, cost 14.52 s 2023-09-05 15:41:29,227 44k INFO ====> Epoch: 8892, cost 14.32 s 2023-09-05 15:41:43,806 44k INFO ====> Epoch: 8893, cost 14.58 s 2023-09-05 15:41:58,506 44k INFO ====> Epoch: 8894, cost 14.70 s 2023-09-05 15:42:13,379 44k INFO ====> Epoch: 8895, cost 14.87 s 2023-09-05 15:42:16,880 44k INFO Train Epoch: 8896 [19%] 2023-09-05 15:42:16,880 44k INFO Losses: [2.2956790924072266, 2.5371198654174805, 8.210884094238281, 14.875173568725586, 0.5082108378410339], step: 186800, lr: 3.289187569807832e-05, reference_loss: 28.42706871032715 2023-09-05 15:42:28,108 44k INFO ====> Epoch: 8896, cost 14.73 s 2023-09-05 15:42:42,751 44k INFO ====> Epoch: 8897, cost 14.64 s 2023-09-05 15:42:57,368 44k INFO ====> Epoch: 8898, cost 14.62 s 2023-09-05 15:43:11,897 44k INFO ====> Epoch: 8899, cost 14.53 s 2023-09-05 15:43:26,294 44k INFO ====> Epoch: 8900, cost 14.40 s 2023-09-05 15:43:40,816 44k INFO ====> Epoch: 8901, cost 14.52 s 2023-09-05 15:43:55,337 44k INFO ====> Epoch: 8902, cost 14.52 s 2023-09-05 15:44:09,880 44k INFO ====> Epoch: 8903, cost 14.54 s 2023-09-05 15:44:24,445 44k INFO ====> Epoch: 8904, cost 14.56 s 2023-09-05 15:44:35,585 44k INFO Train Epoch: 8905 [71%] 2023-09-05 15:44:35,586 44k INFO Losses: [2.1004958152770996, 2.7647697925567627, 9.66321086883545, 14.347021102905273, 0.4175236225128174], step: 187000, lr: 3.2854890834202745e-05, reference_loss: 29.29302215576172 2023-09-05 15:44:39,072 44k INFO ====> Epoch: 8905, cost 14.63 s 2023-09-05 15:44:53,542 44k INFO ====> Epoch: 8906, cost 14.47 s 2023-09-05 15:45:08,159 44k INFO ====> Epoch: 8907, cost 14.62 s 2023-09-05 15:45:22,644 44k INFO ====> Epoch: 8908, cost 14.49 s 2023-09-05 15:45:37,073 44k INFO ====> Epoch: 8909, cost 14.43 s 2023-09-05 15:45:51,508 44k INFO ====> Epoch: 8910, cost 14.43 s 2023-09-05 15:46:05,961 44k INFO ====> Epoch: 8911, cost 14.45 s 2023-09-05 15:46:20,440 44k INFO ====> Epoch: 8912, cost 14.48 s 2023-09-05 15:46:34,699 44k INFO ====> Epoch: 8913, cost 14.26 s 2023-09-05 15:46:49,440 44k INFO ====> Epoch: 8914, cost 14.74 s 2023-09-05 15:46:53,715 44k INFO Train Epoch: 8915 [24%] 2023-09-05 15:46:53,716 44k INFO Losses: [2.153151035308838, 2.640364170074463, 10.101956367492676, 15.783500671386719, 0.6555038094520569], step: 187200, lr: 3.2813845314056416e-05, reference_loss: 31.334476470947266 2023-09-05 15:46:59,253 44k INFO Saving model and optimizer state at iteration 8915 to ./logs\44k\G_187200.pth 2023-09-05 15:46:59,856 44k INFO Saving model and optimizer state at iteration 8915 to ./logs\44k\D_187200.pth 2023-09-05 15:47:01,144 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_184800.pth 2023-09-05 15:47:01,182 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_184800.pth 2023-09-05 15:47:11,534 44k INFO ====> Epoch: 8915, cost 22.09 s 2023-09-05 15:47:25,897 44k INFO ====> Epoch: 8916, cost 14.36 s 2023-09-05 15:47:40,408 44k INFO ====> Epoch: 8917, cost 14.51 s 2023-09-05 15:47:55,114 44k INFO ====> Epoch: 8918, cost 14.71 s 2023-09-05 15:48:09,616 44k INFO ====> Epoch: 8919, cost 14.50 s 2023-09-05 15:48:24,174 44k INFO ====> Epoch: 8920, cost 14.56 s 2023-09-05 15:48:38,668 44k INFO ====> Epoch: 8921, cost 14.49 s 2023-09-05 15:48:53,464 44k INFO ====> Epoch: 8922, cost 14.80 s 2023-09-05 15:49:08,106 44k INFO ====> Epoch: 8923, cost 14.64 s 2023-09-05 15:49:19,972 44k INFO Train Epoch: 8924 [76%] 2023-09-05 15:49:19,973 44k INFO Losses: [2.0716681480407715, 2.8183720111846924, 10.730210304260254, 16.24842071533203, 0.7220491766929626], step: 187400, lr: 3.277694819048356e-05, reference_loss: 32.590721130371094 2023-09-05 15:49:22,732 44k INFO ====> Epoch: 8924, cost 14.63 s 2023-09-05 15:49:37,094 44k INFO ====> Epoch: 8925, cost 14.36 s 2023-09-05 15:49:51,746 44k INFO ====> Epoch: 8926, cost 14.65 s 2023-09-05 15:50:06,532 44k INFO ====> Epoch: 8927, cost 14.79 s 2023-09-05 15:50:20,916 44k INFO ====> Epoch: 8928, cost 14.38 s 2023-09-05 15:50:35,254 44k INFO ====> Epoch: 8929, cost 14.34 s 2023-09-05 15:50:49,695 44k INFO ====> Epoch: 8930, cost 14.44 s 2023-09-05 15:51:04,215 44k INFO ====> Epoch: 8931, cost 14.52 s 2023-09-05 15:51:18,675 44k INFO ====> Epoch: 8932, cost 14.46 s 2023-09-05 15:51:33,248 44k INFO ====> Epoch: 8933, cost 14.57 s 2023-09-05 15:51:38,133 44k INFO Train Epoch: 8934 [29%] 2023-09-05 15:51:38,133 44k INFO Losses: [2.2917063236236572, 2.613828420639038, 10.002201080322266, 15.420738220214844, 0.529647707939148], step: 187600, lr: 3.273600004385673e-05, reference_loss: 30.858121871948242 2023-09-05 15:51:48,082 44k INFO ====> Epoch: 8934, cost 14.83 s 2023-09-05 15:52:02,620 44k INFO ====> Epoch: 8935, cost 14.54 s 2023-09-05 15:52:17,201 44k INFO ====> Epoch: 8936, cost 14.58 s 2023-09-05 15:52:31,520 44k INFO ====> Epoch: 8937, cost 14.32 s 2023-09-05 15:52:46,153 44k INFO ====> Epoch: 8938, cost 14.63 s 2023-09-05 15:53:00,648 44k INFO ====> Epoch: 8939, cost 14.49 s 2023-09-05 15:53:15,218 44k INFO ====> Epoch: 8940, cost 14.57 s 2023-09-05 15:53:29,624 44k INFO ====> Epoch: 8941, cost 14.41 s 2023-09-05 15:53:44,125 44k INFO ====> Epoch: 8942, cost 14.50 s 2023-09-05 15:53:56,871 44k INFO Train Epoch: 8943 [81%] 2023-09-05 15:53:56,871 44k INFO Losses: [2.1713531017303467, 2.787943124771118, 9.399467468261719, 14.540224075317383, 0.6856792569160461], step: 187800, lr: 3.269919045243766e-05, reference_loss: 29.584665298461914 2023-09-05 15:53:59,033 44k INFO ====> Epoch: 8943, cost 14.91 s 2023-09-05 15:54:13,894 44k INFO ====> Epoch: 8944, cost 14.86 s 2023-09-05 15:54:28,381 44k INFO ====> Epoch: 8945, cost 14.49 s 2023-09-05 15:54:42,780 44k INFO ====> Epoch: 8946, cost 14.40 s 2023-09-05 15:54:57,214 44k INFO ====> Epoch: 8947, cost 14.43 s 2023-09-05 15:55:11,704 44k INFO ====> Epoch: 8948, cost 14.49 s 2023-09-05 15:55:25,977 44k INFO ====> Epoch: 8949, cost 14.27 s 2023-09-05 15:55:40,236 44k INFO ====> Epoch: 8950, cost 14.26 s 2023-09-05 15:55:54,814 44k INFO ====> Epoch: 8951, cost 14.58 s 2023-09-05 15:56:09,584 44k INFO ====> Epoch: 8952, cost 14.77 s 2023-09-05 15:56:15,198 44k INFO Train Epoch: 8953 [33%] 2023-09-05 15:56:15,198 44k INFO Losses: [2.1244406700134277, 2.707730293273926, 7.926346778869629, 14.422992706298828, 0.6149790287017822], step: 188000, lr: 3.26583394483282e-05, reference_loss: 27.796489715576172 2023-09-05 15:56:20,672 44k INFO Saving model and optimizer state at iteration 8953 to ./logs\44k\G_188000.pth 2023-09-05 15:56:21,220 44k INFO Saving model and optimizer state at iteration 8953 to ./logs\44k\D_188000.pth 2023-09-05 15:56:22,926 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_185600.pth 2023-09-05 15:56:22,958 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_185600.pth 2023-09-05 15:56:31,867 44k INFO ====> Epoch: 8953, cost 22.28 s 2023-09-05 15:56:46,428 44k INFO ====> Epoch: 8954, cost 14.56 s 2023-09-05 15:57:01,097 44k INFO ====> Epoch: 8955, cost 14.67 s 2023-09-05 15:57:15,655 44k INFO ====> Epoch: 8956, cost 14.56 s 2023-09-05 15:57:30,117 44k INFO ====> Epoch: 8957, cost 14.46 s 2023-09-05 15:57:44,656 44k INFO ====> Epoch: 8958, cost 14.54 s 2023-09-05 15:57:59,223 44k INFO ====> Epoch: 8959, cost 14.57 s 2023-09-05 15:58:13,753 44k INFO ====> Epoch: 8960, cost 14.53 s 2023-09-05 15:58:28,421 44k INFO ====> Epoch: 8961, cost 14.67 s 2023-09-05 15:58:41,718 44k INFO Train Epoch: 8962 [86%] 2023-09-05 15:58:41,718 44k INFO Losses: [2.3353590965270996, 2.5534422397613525, 11.00981330871582, 15.239510536193848, 0.7039814591407776], step: 188200, lr: 3.262161718140775e-05, reference_loss: 31.84210777282715 2023-09-05 15:58:43,131 44k INFO ====> Epoch: 8962, cost 14.71 s 2023-09-05 15:58:57,988 44k INFO ====> Epoch: 8963, cost 14.86 s 2023-09-05 15:59:12,588 44k INFO ====> Epoch: 8964, cost 14.60 s 2023-09-05 15:59:27,085 44k INFO ====> Epoch: 8965, cost 14.50 s 2023-09-05 15:59:41,474 44k INFO ====> Epoch: 8966, cost 14.39 s 2023-09-05 15:59:56,239 44k INFO ====> Epoch: 8967, cost 14.76 s 2023-09-05 16:00:10,957 44k INFO ====> Epoch: 8968, cost 14.72 s 2023-09-05 16:00:25,405 44k INFO ====> Epoch: 8969, cost 14.45 s 2023-09-05 16:00:39,743 44k INFO ====> Epoch: 8970, cost 14.34 s 2023-09-05 16:00:54,421 44k INFO ====> Epoch: 8971, cost 14.68 s 2023-09-05 16:01:00,736 44k INFO Train Epoch: 8972 [38%] 2023-09-05 16:01:00,737 44k INFO Losses: [2.2389776706695557, 2.6304831504821777, 10.76293659210205, 15.037296295166016, 0.5784162282943726], step: 188400, lr: 3.258086308936154e-05, reference_loss: 31.248109817504883 2023-09-05 16:01:09,290 44k INFO ====> Epoch: 8972, cost 14.87 s 2023-09-05 16:01:23,696 44k INFO ====> Epoch: 8973, cost 14.41 s 2023-09-05 16:01:38,177 44k INFO ====> Epoch: 8974, cost 14.48 s 2023-09-05 16:01:52,810 44k INFO ====> Epoch: 8975, cost 14.63 s 2023-09-05 16:02:07,646 44k INFO ====> Epoch: 8976, cost 14.84 s 2023-09-05 16:02:21,936 44k INFO ====> Epoch: 8977, cost 14.29 s 2023-09-05 16:02:36,308 44k INFO ====> Epoch: 8978, cost 14.37 s 2023-09-05 16:02:50,799 44k INFO ====> Epoch: 8979, cost 14.49 s 2023-09-05 16:03:05,304 44k INFO ====> Epoch: 8980, cost 14.50 s 2023-09-05 16:03:19,219 44k INFO Train Epoch: 8981 [90%] 2023-09-05 16:03:19,220 44k INFO Losses: [2.0532593727111816, 2.695679187774658, 11.060115814208984, 16.184770584106445, 0.4025675058364868], step: 188600, lr: 3.254422793977719e-05, reference_loss: 32.396392822265625 2023-09-05 16:03:19,877 44k INFO ====> Epoch: 8981, cost 14.57 s 2023-09-05 16:03:34,433 44k INFO ====> Epoch: 8982, cost 14.56 s 2023-09-05 16:03:48,869 44k INFO ====> Epoch: 8983, cost 14.44 s 2023-09-05 16:04:03,450 44k INFO ====> Epoch: 8984, cost 14.58 s 2023-09-05 16:04:17,797 44k INFO ====> Epoch: 8985, cost 14.35 s 2023-09-05 16:04:32,466 44k INFO ====> Epoch: 8986, cost 14.67 s 2023-09-05 16:04:47,049 44k INFO ====> Epoch: 8987, cost 14.58 s 2023-09-05 16:05:01,598 44k INFO ====> Epoch: 8988, cost 14.55 s 2023-09-05 16:05:15,977 44k INFO ====> Epoch: 8989, cost 14.38 s 2023-09-05 16:05:30,315 44k INFO ====> Epoch: 8990, cost 14.34 s 2023-09-05 16:05:37,247 44k INFO Train Epoch: 8991 [43%] 2023-09-05 16:05:37,248 44k INFO Losses: [2.4536783695220947, 2.5343918800354004, 10.363767623901367, 15.224533081054688, 0.6966065168380737], step: 188800, lr: 3.250357052988684e-05, reference_loss: 31.27297592163086 2023-09-05 16:05:42,816 44k INFO Saving model and optimizer state at iteration 8991 to ./logs\44k\G_188800.pth 2023-09-05 16:05:43,360 44k INFO Saving model and optimizer state at iteration 8991 to ./logs\44k\D_188800.pth 2023-09-05 16:05:44,570 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_186400.pth 2023-09-05 16:05:44,606 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_186400.pth 2023-09-05 16:05:52,176 44k INFO ====> Epoch: 8991, cost 21.86 s 2023-09-05 16:06:07,062 44k INFO ====> Epoch: 8992, cost 14.89 s 2023-09-05 16:06:21,505 44k INFO ====> Epoch: 8993, cost 14.44 s 2023-09-05 16:06:36,052 44k INFO ====> Epoch: 8994, cost 14.55 s 2023-09-05 16:06:50,650 44k INFO ====> Epoch: 8995, cost 14.60 s 2023-09-05 16:07:05,293 44k INFO ====> Epoch: 8996, cost 14.64 s 2023-09-05 16:07:19,981 44k INFO ====> Epoch: 8997, cost 14.69 s 2023-09-05 16:07:34,413 44k INFO ====> Epoch: 8998, cost 14.43 s 2023-09-05 16:07:49,030 44k INFO ====> Epoch: 8999, cost 14.62 s 2023-09-05 16:08:03,893 44k INFO Train Epoch: 9000 [95%] 2023-09-05 16:08:03,894 44k INFO Losses: [2.5881152153015137, 2.140871524810791, 6.889391899108887, 13.45702075958252, 0.5454020500183105], step: 189000, lr: 3.24670222909675e-05, reference_loss: 25.62080192565918 2023-09-05 16:08:04,234 44k INFO ====> Epoch: 9000, cost 15.20 s 2023-09-05 16:08:18,761 44k INFO ====> Epoch: 9001, cost 14.53 s 2023-09-05 16:08:33,230 44k INFO ====> Epoch: 9002, cost 14.47 s 2023-09-05 16:08:47,852 44k INFO ====> Epoch: 9003, cost 14.62 s 2023-09-05 16:09:02,494 44k INFO ====> Epoch: 9004, cost 14.64 s 2023-09-05 16:09:17,017 44k INFO ====> Epoch: 9005, cost 14.52 s 2023-09-05 16:09:31,240 44k INFO ====> Epoch: 9006, cost 14.22 s 2023-09-05 16:09:45,846 44k INFO ====> Epoch: 9007, cost 14.61 s 2023-09-05 16:10:00,492 44k INFO ====> Epoch: 9008, cost 14.65 s 2023-09-05 16:10:15,151 44k INFO ====> Epoch: 9009, cost 14.66 s 2023-09-05 16:10:22,831 44k INFO Train Epoch: 9010 [48%] 2023-09-05 16:10:22,832 44k INFO Losses: [2.170727014541626, 2.7370688915252686, 11.799120903015137, 16.975492477416992, 0.6020951271057129], step: 189200, lr: 3.242646133387103e-05, reference_loss: 34.284507751464844 2023-09-05 16:10:29,791 44k INFO ====> Epoch: 9010, cost 14.64 s 2023-09-05 16:10:44,365 44k INFO ====> Epoch: 9011, cost 14.57 s 2023-09-05 16:10:59,000 44k INFO ====> Epoch: 9012, cost 14.63 s 2023-09-05 16:11:13,514 44k INFO ====> Epoch: 9013, cost 14.51 s 2023-09-05 16:11:27,935 44k INFO ====> Epoch: 9014, cost 14.42 s 2023-09-05 16:11:42,556 44k INFO ====> Epoch: 9015, cost 14.62 s 2023-09-05 16:11:57,256 44k INFO ====> Epoch: 9016, cost 14.70 s 2023-09-05 16:12:11,941 44k INFO ====> Epoch: 9017, cost 14.68 s 2023-09-05 16:12:26,364 44k INFO ====> Epoch: 9018, cost 14.42 s 2023-09-05 16:12:40,877 44k INFO ====> Epoch: 9019, cost 14.51 s 2023-09-05 16:12:41,610 44k INFO Train Epoch: 9020 [0%] 2023-09-05 16:12:41,610 44k INFO Losses: [1.992862582206726, 3.1367857456207275, 10.980500221252441, 15.176267623901367, 0.704572319984436], step: 189400, lr: 3.238595104946102e-05, reference_loss: 31.990989685058594 2023-09-05 16:12:55,703 44k INFO ====> Epoch: 9020, cost 14.83 s 2023-09-05 16:13:10,286 44k INFO ====> Epoch: 9021, cost 14.58 s 2023-09-05 16:13:24,738 44k INFO ====> Epoch: 9022, cost 14.45 s 2023-09-05 16:13:39,051 44k INFO ====> Epoch: 9023, cost 14.31 s 2023-09-05 16:13:53,753 44k INFO ====> Epoch: 9024, cost 14.70 s 2023-09-05 16:14:08,412 44k INFO ====> Epoch: 9025, cost 14.66 s 2023-09-05 16:14:22,828 44k INFO ====> Epoch: 9026, cost 14.42 s 2023-09-05 16:14:37,579 44k INFO ====> Epoch: 9027, cost 14.75 s 2023-09-05 16:14:52,057 44k INFO ====> Epoch: 9028, cost 14.48 s 2023-09-05 16:15:00,504 44k INFO Train Epoch: 9029 [52%] 2023-09-05 16:15:00,504 44k INFO Losses: [1.9885417222976685, 2.9205286502838135, 11.121320724487305, 14.754817008972168, 0.6464874148368835], step: 189600, lr: 3.234953506631551e-05, reference_loss: 31.431697845458984 2023-09-05 16:15:06,049 44k INFO Saving model and optimizer state at iteration 9029 to ./logs\44k\G_189600.pth 2023-09-05 16:15:06,644 44k INFO Saving model and optimizer state at iteration 9029 to ./logs\44k\D_189600.pth 2023-09-05 16:15:07,768 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_187200.pth 2023-09-05 16:15:07,806 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_187200.pth 2023-09-05 16:15:13,775 44k INFO ====> Epoch: 9029, cost 21.72 s 2023-09-05 16:15:28,051 44k INFO ====> Epoch: 9030, cost 14.28 s 2023-09-05 16:15:42,586 44k INFO ====> Epoch: 9031, cost 14.53 s 2023-09-05 16:15:57,167 44k INFO ====> Epoch: 9032, cost 14.58 s 2023-09-05 16:16:11,756 44k INFO ====> Epoch: 9033, cost 14.59 s 2023-09-05 16:16:26,315 44k INFO ====> Epoch: 9034, cost 14.56 s 2023-09-05 16:16:40,630 44k INFO ====> Epoch: 9035, cost 14.31 s 2023-09-05 16:16:55,342 44k INFO ====> Epoch: 9036, cost 14.71 s 2023-09-05 16:17:09,999 44k INFO ====> Epoch: 9037, cost 14.66 s 2023-09-05 16:17:24,498 44k INFO ====> Epoch: 9038, cost 14.50 s 2023-09-05 16:17:25,860 44k INFO Train Epoch: 9039 [5%] 2023-09-05 16:17:25,860 44k INFO Losses: [2.124925136566162, 2.8051257133483887, 9.66774845123291, 13.877370834350586, 0.6310234665870667], step: 189800, lr: 3.2309120885669176e-05, reference_loss: 29.10619354248047 2023-09-05 16:17:39,200 44k INFO ====> Epoch: 9039, cost 14.70 s 2023-09-05 16:17:54,026 44k INFO ====> Epoch: 9040, cost 14.83 s 2023-09-05 16:18:08,685 44k INFO ====> Epoch: 9041, cost 14.66 s 2023-09-05 16:18:23,198 44k INFO ====> Epoch: 9042, cost 14.51 s 2023-09-05 16:18:37,569 44k INFO ====> Epoch: 9043, cost 14.37 s 2023-09-05 16:18:52,085 44k INFO ====> Epoch: 9044, cost 14.52 s 2023-09-05 16:19:06,734 44k INFO ====> Epoch: 9045, cost 14.65 s 2023-09-05 16:19:21,163 44k INFO ====> Epoch: 9046, cost 14.43 s 2023-09-05 16:19:35,552 44k INFO ====> Epoch: 9047, cost 14.39 s 2023-09-05 16:19:44,746 44k INFO Train Epoch: 9048 [57%] 2023-09-05 16:19:44,746 44k INFO Losses: [2.1285853385925293, 3.0302112102508545, 10.609086990356445, 15.987114906311035, 0.4876890182495117], step: 190000, lr: 3.227279129325357e-05, reference_loss: 32.2426872253418 2023-09-05 16:19:50,395 44k INFO ====> Epoch: 9048, cost 14.84 s 2023-09-05 16:20:05,115 44k INFO ====> Epoch: 9049, cost 14.72 s 2023-09-05 16:20:19,726 44k INFO ====> Epoch: 9050, cost 14.61 s 2023-09-05 16:20:34,285 44k INFO ====> Epoch: 9051, cost 14.56 s 2023-09-05 16:20:48,717 44k INFO ====> Epoch: 9052, cost 14.43 s 2023-09-05 16:21:03,346 44k INFO ====> Epoch: 9053, cost 14.63 s 2023-09-05 16:21:17,702 44k INFO ====> Epoch: 9054, cost 14.36 s 2023-09-05 16:21:32,022 44k INFO ====> Epoch: 9055, cost 14.32 s 2023-09-05 16:21:46,858 44k INFO ====> Epoch: 9056, cost 14.84 s 2023-09-05 16:22:01,473 44k INFO ====> Epoch: 9057, cost 14.61 s 2023-09-05 16:22:03,604 44k INFO Train Epoch: 9058 [10%] 2023-09-05 16:22:03,604 44k INFO Losses: [2.143352508544922, 2.7902536392211914, 8.145844459533691, 14.255001068115234, 0.583569347858429], step: 190200, lr: 3.2232472988381086e-05, reference_loss: 27.918020248413086 2023-09-05 16:22:16,351 44k INFO ====> Epoch: 9058, cost 14.88 s 2023-09-05 16:22:30,930 44k INFO ====> Epoch: 9059, cost 14.58 s 2023-09-05 16:22:45,468 44k INFO ====> Epoch: 9060, cost 14.54 s 2023-09-05 16:23:00,169 44k INFO ====> Epoch: 9061, cost 14.70 s 2023-09-05 16:23:14,692 44k INFO ====> Epoch: 9062, cost 14.52 s 2023-09-05 16:23:28,918 44k INFO ====> Epoch: 9063, cost 14.22 s 2023-09-05 16:23:43,490 44k INFO ====> Epoch: 9064, cost 14.57 s 2023-09-05 16:23:58,153 44k INFO ====> Epoch: 9065, cost 14.66 s 2023-09-05 16:24:12,773 44k INFO ====> Epoch: 9066, cost 14.62 s 2023-09-05 16:24:22,659 44k INFO Train Epoch: 9067 [62%] 2023-09-05 16:24:22,660 44k INFO Losses: [2.066141366958618, 2.912421941757202, 9.13373851776123, 14.795612335205078, 0.5696437358856201], step: 190400, lr: 3.219622958174806e-05, reference_loss: 29.477556228637695 2023-09-05 16:24:28,174 44k INFO Saving model and optimizer state at iteration 9067 to ./logs\44k\G_190400.pth 2023-09-05 16:24:28,819 44k INFO Saving model and optimizer state at iteration 9067 to ./logs\44k\D_190400.pth 2023-09-05 16:24:29,991 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_188000.pth 2023-09-05 16:24:30,027 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_188000.pth 2023-09-05 16:24:34,608 44k INFO ====> Epoch: 9067, cost 21.83 s 2023-09-05 16:24:49,179 44k INFO ====> Epoch: 9068, cost 14.57 s 2023-09-05 16:25:03,778 44k INFO ====> Epoch: 9069, cost 14.60 s 2023-09-05 16:25:18,115 44k INFO ====> Epoch: 9070, cost 14.34 s 2023-09-05 16:25:32,550 44k INFO ====> Epoch: 9071, cost 14.44 s 2023-09-05 16:25:46,961 44k INFO ====> Epoch: 9072, cost 14.41 s 2023-09-05 16:26:01,578 44k INFO ====> Epoch: 9073, cost 14.62 s 2023-09-05 16:26:16,054 44k INFO ====> Epoch: 9074, cost 14.48 s 2023-09-05 16:26:30,470 44k INFO ====> Epoch: 9075, cost 14.42 s 2023-09-05 16:26:44,938 44k INFO ====> Epoch: 9076, cost 14.47 s 2023-09-05 16:26:47,704 44k INFO Train Epoch: 9077 [14%] 2023-09-05 16:26:47,705 44k INFO Losses: [2.075577735900879, 2.8244118690490723, 10.002901077270508, 16.112537384033203, 0.7046449780464172], step: 190600, lr: 3.215600692520045e-05, reference_loss: 31.720073699951172 2023-09-05 16:26:59,692 44k INFO ====> Epoch: 9077, cost 14.75 s 2023-09-05 16:27:14,243 44k INFO ====> Epoch: 9078, cost 14.55 s 2023-09-05 16:27:28,660 44k INFO ====> Epoch: 9079, cost 14.42 s 2023-09-05 16:27:43,165 44k INFO ====> Epoch: 9080, cost 14.50 s 2023-09-05 16:27:57,928 44k INFO ====> Epoch: 9081, cost 14.76 s 2023-09-05 16:28:12,323 44k INFO ====> Epoch: 9082, cost 14.39 s 2023-09-05 16:28:26,925 44k INFO ====> Epoch: 9083, cost 14.60 s 2023-09-05 16:28:41,362 44k INFO ====> Epoch: 9084, cost 14.44 s 2023-09-05 16:28:56,080 44k INFO ====> Epoch: 9085, cost 14.72 s 2023-09-05 16:29:06,748 44k INFO Train Epoch: 9086 [67%] 2023-09-05 16:29:06,748 44k INFO Losses: [2.1376309394836426, 2.810281276702881, 11.358901977539062, 15.610014915466309, 0.5330187082290649], step: 190800, lr: 3.211984949988888e-05, reference_loss: 32.44984817504883 2023-09-05 16:29:10,991 44k INFO ====> Epoch: 9086, cost 14.91 s 2023-09-05 16:29:25,421 44k INFO ====> Epoch: 9087, cost 14.43 s 2023-09-05 16:29:39,931 44k INFO ====> Epoch: 9088, cost 14.51 s 2023-09-05 16:29:54,700 44k INFO ====> Epoch: 9089, cost 14.77 s 2023-09-05 16:30:09,425 44k INFO ====> Epoch: 9090, cost 14.73 s 2023-09-05 16:30:23,981 44k INFO ====> Epoch: 9091, cost 14.56 s 2023-09-05 16:30:38,577 44k INFO ====> Epoch: 9092, cost 14.60 s 2023-09-05 16:30:53,300 44k INFO ====> Epoch: 9093, cost 14.72 s 2023-09-05 16:31:07,989 44k INFO ====> Epoch: 9094, cost 14.69 s 2023-09-05 16:31:22,450 44k INFO ====> Epoch: 9095, cost 14.46 s 2023-09-05 16:31:25,895 44k INFO Train Epoch: 9096 [19%] 2023-09-05 16:31:25,896 44k INFO Losses: [2.1637122631073, 2.7464425563812256, 11.504044532775879, 15.175601959228516, 0.5410160422325134], step: 191000, lr: 3.207972226475674e-05, reference_loss: 32.13081741333008 2023-09-05 16:31:37,178 44k INFO ====> Epoch: 9096, cost 14.73 s 2023-09-05 16:31:51,802 44k INFO ====> Epoch: 9097, cost 14.62 s 2023-09-05 16:32:06,520 44k INFO ====> Epoch: 9098, cost 14.72 s 2023-09-05 16:32:20,982 44k INFO ====> Epoch: 9099, cost 14.46 s 2023-09-05 16:32:35,479 44k INFO ====> Epoch: 9100, cost 14.50 s 2023-09-05 16:32:49,911 44k INFO ====> Epoch: 9101, cost 14.43 s 2023-09-05 16:33:04,411 44k INFO ====> Epoch: 9102, cost 14.50 s 2023-09-05 16:33:18,798 44k INFO ====> Epoch: 9103, cost 14.39 s 2023-09-05 16:33:33,241 44k INFO ====> Epoch: 9104, cost 14.44 s 2023-09-05 16:33:44,296 44k INFO Train Epoch: 9105 [71%] 2023-09-05 16:33:44,297 44k INFO Losses: [2.0661962032318115, 2.8893423080444336, 9.239612579345703, 15.058280944824219, 0.619179904460907], step: 191200, lr: 3.204365061679055e-05, reference_loss: 29.87261390686035 2023-09-05 16:33:49,804 44k INFO Saving model and optimizer state at iteration 9105 to ./logs\44k\G_191200.pth 2023-09-05 16:33:50,493 44k INFO Saving model and optimizer state at iteration 9105 to ./logs\44k\D_191200.pth 2023-09-05 16:33:51,786 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_188800.pth 2023-09-05 16:33:51,822 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_188800.pth 2023-09-05 16:33:55,109 44k INFO ====> Epoch: 9105, cost 21.87 s 2023-09-05 16:34:09,809 44k INFO ====> Epoch: 9106, cost 14.70 s 2023-09-05 16:34:24,286 44k INFO ====> Epoch: 9107, cost 14.48 s 2023-09-05 16:34:38,658 44k INFO ====> Epoch: 9108, cost 14.37 s 2023-09-05 16:34:53,272 44k INFO ====> Epoch: 9109, cost 14.61 s 2023-09-05 16:35:07,814 44k INFO ====> Epoch: 9110, cost 14.54 s 2023-09-05 16:35:22,408 44k INFO ====> Epoch: 9111, cost 14.59 s 2023-09-05 16:35:36,695 44k INFO ====> Epoch: 9112, cost 14.29 s 2023-09-05 16:35:51,316 44k INFO ====> Epoch: 9113, cost 14.62 s 2023-09-05 16:36:05,916 44k INFO ====> Epoch: 9114, cost 14.60 s 2023-09-05 16:36:10,111 44k INFO Train Epoch: 9115 [24%] 2023-09-05 16:36:10,111 44k INFO Losses: [2.1999104022979736, 2.645766496658325, 8.36280345916748, 13.765469551086426, 0.5890082716941833], step: 191400, lr: 3.20036185767028e-05, reference_loss: 27.562957763671875 2023-09-05 16:36:20,699 44k INFO ====> Epoch: 9115, cost 14.78 s 2023-09-05 16:36:35,154 44k INFO ====> Epoch: 9116, cost 14.45 s 2023-09-05 16:36:49,821 44k INFO ====> Epoch: 9117, cost 14.67 s 2023-09-05 16:37:04,579 44k INFO ====> Epoch: 9118, cost 14.76 s 2023-09-05 16:37:19,100 44k INFO ====> Epoch: 9119, cost 14.52 s 2023-09-05 16:37:33,340 44k INFO ====> Epoch: 9120, cost 14.24 s 2023-09-05 16:37:47,820 44k INFO ====> Epoch: 9121, cost 14.48 s 2023-09-05 16:38:02,607 44k INFO ====> Epoch: 9122, cost 14.79 s 2023-09-05 16:38:17,115 44k INFO ====> Epoch: 9123, cost 14.51 s 2023-09-05 16:38:28,932 44k INFO Train Epoch: 9124 [76%] 2023-09-05 16:38:28,932 44k INFO Losses: [2.142029047012329, 2.662780523300171, 10.882620811462402, 16.150306701660156, 0.5303435921669006], step: 191600, lr: 3.196763250258985e-05, reference_loss: 32.368080139160156 2023-09-05 16:38:31,819 44k INFO ====> Epoch: 9124, cost 14.70 s 2023-09-05 16:38:46,242 44k INFO ====> Epoch: 9125, cost 14.42 s 2023-09-05 16:39:00,807 44k INFO ====> Epoch: 9126, cost 14.57 s 2023-09-05 16:39:15,167 44k INFO ====> Epoch: 9127, cost 14.36 s 2023-09-05 16:39:29,413 44k INFO ====> Epoch: 9128, cost 14.25 s 2023-09-05 16:39:43,831 44k INFO ====> Epoch: 9129, cost 14.42 s 2023-09-05 16:39:58,440 44k INFO ====> Epoch: 9130, cost 14.61 s 2023-09-05 16:40:12,795 44k INFO ====> Epoch: 9131, cost 14.35 s 2023-09-05 16:40:27,209 44k INFO ====> Epoch: 9132, cost 14.41 s 2023-09-05 16:40:41,666 44k INFO ====> Epoch: 9133, cost 14.46 s 2023-09-05 16:40:46,635 44k INFO Train Epoch: 9134 [29%] 2023-09-05 16:40:46,635 44k INFO Losses: [1.991288185119629, 3.036118268966675, 11.26374340057373, 13.302225112915039, 0.47601932287216187], step: 191800, lr: 3.192769543171243e-05, reference_loss: 30.069393157958984 2023-09-05 16:40:56,783 44k INFO ====> Epoch: 9134, cost 15.12 s 2023-09-05 16:41:11,167 44k INFO ====> Epoch: 9135, cost 14.38 s 2023-09-05 16:41:25,462 44k INFO ====> Epoch: 9136, cost 14.29 s 2023-09-05 16:41:39,693 44k INFO ====> Epoch: 9137, cost 14.23 s 2023-09-05 16:41:54,322 44k INFO ====> Epoch: 9138, cost 14.63 s 2023-09-05 16:42:09,106 44k INFO ====> Epoch: 9139, cost 14.78 s 2023-09-05 16:42:23,757 44k INFO ====> Epoch: 9140, cost 14.65 s 2023-09-05 16:42:38,245 44k INFO ====> Epoch: 9141, cost 14.49 s 2023-09-05 16:42:52,849 44k INFO ====> Epoch: 9142, cost 14.60 s 2023-09-05 16:43:05,652 44k INFO Train Epoch: 9143 [81%] 2023-09-05 16:43:05,652 44k INFO Losses: [2.1442127227783203, 2.7215161323547363, 10.839268684387207, 14.909489631652832, 0.6185978651046753], step: 192000, lr: 3.189179472844327e-05, reference_loss: 31.233083724975586 2023-09-05 16:43:11,262 44k INFO Saving model and optimizer state at iteration 9143 to ./logs\44k\G_192000.pth 2023-09-05 16:43:11,803 44k INFO Saving model and optimizer state at iteration 9143 to ./logs\44k\D_192000.pth 2023-09-05 16:43:13,632 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_189600.pth 2023-09-05 16:43:13,668 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_189600.pth 2023-09-05 16:43:15,561 44k INFO ====> Epoch: 9143, cost 22.71 s 2023-09-05 16:43:30,084 44k INFO ====> Epoch: 9144, cost 14.52 s 2023-09-05 16:43:44,616 44k INFO ====> Epoch: 9145, cost 14.53 s 2023-09-05 16:43:59,181 44k INFO ====> Epoch: 9146, cost 14.57 s 2023-09-05 16:44:13,928 44k INFO ====> Epoch: 9147, cost 14.75 s 2023-09-05 16:44:28,274 44k INFO ====> Epoch: 9148, cost 14.35 s 2023-09-05 16:44:42,789 44k INFO ====> Epoch: 9149, cost 14.52 s 2023-09-05 16:44:57,500 44k INFO ====> Epoch: 9150, cost 14.71 s 2023-09-05 16:45:12,078 44k INFO ====> Epoch: 9151, cost 14.58 s 2023-09-05 16:45:26,469 44k INFO ====> Epoch: 9152, cost 14.39 s 2023-09-05 16:45:32,146 44k INFO Train Epoch: 9153 [33%] 2023-09-05 16:45:32,147 44k INFO Losses: [2.2211735248565674, 2.4684901237487793, 8.55333137512207, 14.128588676452637, 0.5899958610534668], step: 192200, lr: 3.185195240147787e-05, reference_loss: 27.961580276489258 2023-09-05 16:45:41,193 44k INFO ====> Epoch: 9153, cost 14.72 s 2023-09-05 16:45:55,838 44k INFO ====> Epoch: 9154, cost 14.65 s 2023-09-05 16:46:10,249 44k INFO ====> Epoch: 9155, cost 14.41 s 2023-09-05 16:46:24,878 44k INFO ====> Epoch: 9156, cost 14.63 s 2023-09-05 16:46:39,224 44k INFO ====> Epoch: 9157, cost 14.35 s 2023-09-05 16:46:53,938 44k INFO ====> Epoch: 9158, cost 14.71 s 2023-09-05 16:47:08,534 44k INFO ====> Epoch: 9159, cost 14.60 s 2023-09-05 16:47:22,995 44k INFO ====> Epoch: 9160, cost 14.46 s 2023-09-05 16:47:37,348 44k INFO ====> Epoch: 9161, cost 14.35 s 2023-09-05 16:47:50,914 44k INFO Train Epoch: 9162 [86%] 2023-09-05 16:47:50,914 44k INFO Losses: [2.0759007930755615, 3.0061466693878174, 9.236591339111328, 14.590930938720703, 0.5314332842826843], step: 192400, lr: 3.18161368665247e-05, reference_loss: 29.441001892089844 2023-09-05 16:47:52,267 44k INFO ====> Epoch: 9162, cost 14.92 s 2023-09-05 16:48:07,185 44k INFO ====> Epoch: 9163, cost 14.92 s 2023-09-05 16:48:21,623 44k INFO ====> Epoch: 9164, cost 14.44 s 2023-09-05 16:48:36,131 44k INFO ====> Epoch: 9165, cost 14.51 s 2023-09-05 16:48:50,645 44k INFO ====> Epoch: 9166, cost 14.51 s 2023-09-05 16:49:05,283 44k INFO ====> Epoch: 9167, cost 14.64 s 2023-09-05 16:49:19,615 44k INFO ====> Epoch: 9168, cost 14.33 s 2023-09-05 16:49:33,922 44k INFO ====> Epoch: 9169, cost 14.31 s 2023-09-05 16:49:48,514 44k INFO ====> Epoch: 9170, cost 14.59 s 2023-09-05 16:50:03,184 44k INFO ====> Epoch: 9171, cost 14.67 s 2023-09-05 16:50:09,516 44k INFO Train Epoch: 9172 [38%] 2023-09-05 16:50:09,516 44k INFO Losses: [2.2675442695617676, 2.5519869327545166, 9.708918571472168, 15.204319953918457, 0.43986985087394714], step: 192600, lr: 3.1776389058707494e-05, reference_loss: 30.172637939453125 2023-09-05 16:50:17,996 44k INFO ====> Epoch: 9172, cost 14.81 s 2023-09-05 16:50:32,516 44k INFO ====> Epoch: 9173, cost 14.52 s 2023-09-05 16:50:47,132 44k INFO ====> Epoch: 9174, cost 14.62 s 2023-09-05 16:51:01,643 44k INFO ====> Epoch: 9175, cost 14.51 s 2023-09-05 16:51:16,244 44k INFO ====> Epoch: 9176, cost 14.60 s 2023-09-05 16:51:30,694 44k INFO ====> Epoch: 9177, cost 14.45 s 2023-09-05 16:51:45,008 44k INFO ====> Epoch: 9178, cost 14.31 s 2023-09-05 16:51:59,688 44k INFO ====> Epoch: 9179, cost 14.68 s 2023-09-05 16:52:14,308 44k INFO ====> Epoch: 9180, cost 14.62 s 2023-09-05 16:52:28,263 44k INFO Train Epoch: 9181 [90%] 2023-09-05 16:52:28,263 44k INFO Losses: [2.210301399230957, 2.617112874984741, 9.055296897888184, 14.920746803283691, 0.6975181698799133], step: 192800, lr: 3.1740658490022944e-05, reference_loss: 29.5009765625 2023-09-05 16:52:33,752 44k INFO Saving model and optimizer state at iteration 9181 to ./logs\44k\G_192800.pth 2023-09-05 16:52:34,431 44k INFO Saving model and optimizer state at iteration 9181 to ./logs\44k\D_192800.pth 2023-09-05 16:52:35,707 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_190400.pth 2023-09-05 16:52:35,743 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_190400.pth 2023-09-05 16:52:36,222 44k INFO ====> Epoch: 9181, cost 21.91 s 2023-09-05 16:52:50,874 44k INFO ====> Epoch: 9182, cost 14.65 s 2023-09-05 16:53:05,532 44k INFO ====> Epoch: 9183, cost 14.66 s 2023-09-05 16:53:20,010 44k INFO ====> Epoch: 9184, cost 14.48 s 2023-09-05 16:53:34,601 44k INFO ====> Epoch: 9185, cost 14.59 s 2023-09-05 16:53:49,165 44k INFO ====> Epoch: 9186, cost 14.56 s 2023-09-05 16:54:03,875 44k INFO ====> Epoch: 9187, cost 14.71 s 2023-09-05 16:54:18,493 44k INFO ====> Epoch: 9188, cost 14.62 s 2023-09-05 16:54:32,970 44k INFO ====> Epoch: 9189, cost 14.48 s 2023-09-05 16:54:47,491 44k INFO ====> Epoch: 9190, cost 14.52 s 2023-09-05 16:54:54,497 44k INFO Train Epoch: 9191 [43%] 2023-09-05 16:54:54,498 44k INFO Losses: [2.2224621772766113, 2.7853002548217773, 10.642618179321289, 15.849472045898438, 0.5548242330551147], step: 193000, lr: 3.1701004977123314e-05, reference_loss: 32.0546760559082 2023-09-05 16:55:02,382 44k INFO ====> Epoch: 9191, cost 14.89 s 2023-09-05 16:55:16,813 44k INFO ====> Epoch: 9192, cost 14.43 s 2023-09-05 16:55:31,259 44k INFO ====> Epoch: 9193, cost 14.45 s 2023-09-05 16:55:45,685 44k INFO ====> Epoch: 9194, cost 14.43 s 2023-09-05 16:56:00,208 44k INFO ====> Epoch: 9195, cost 14.52 s 2023-09-05 16:56:14,651 44k INFO ====> Epoch: 9196, cost 14.44 s 2023-09-05 16:56:29,358 44k INFO ====> Epoch: 9197, cost 14.71 s 2023-09-05 16:56:44,047 44k INFO ====> Epoch: 9198, cost 14.69 s 2023-09-05 16:56:58,622 44k INFO ====> Epoch: 9199, cost 14.58 s 2023-09-05 16:57:13,173 44k INFO Train Epoch: 9200 [95%] 2023-09-05 16:57:13,174 44k INFO Losses: [1.66563081741333, 3.1939220428466797, 11.1602144241333, 15.239815711975098, 0.19650955498218536], step: 193200, lr: 3.166535917313937e-05, reference_loss: 31.456092834472656 2023-09-05 16:57:13,436 44k INFO ====> Epoch: 9200, cost 14.81 s 2023-09-05 16:57:27,823 44k INFO ====> Epoch: 9201, cost 14.39 s 2023-09-05 16:57:42,179 44k INFO ====> Epoch: 9202, cost 14.36 s 2023-09-05 16:57:56,770 44k INFO ====> Epoch: 9203, cost 14.59 s 2023-09-05 16:58:11,332 44k INFO ====> Epoch: 9204, cost 14.56 s 2023-09-05 16:58:25,934 44k INFO ====> Epoch: 9205, cost 14.60 s 2023-09-05 16:58:40,785 44k INFO ====> Epoch: 9206, cost 14.85 s 2023-09-05 16:58:55,483 44k INFO ====> Epoch: 9207, cost 14.70 s 2023-09-05 16:59:09,976 44k INFO ====> Epoch: 9208, cost 14.49 s 2023-09-05 16:59:24,297 44k INFO ====> Epoch: 9209, cost 14.32 s 2023-09-05 16:59:31,856 44k INFO Train Epoch: 9210 [48%] 2023-09-05 16:59:31,857 44k INFO Losses: [2.0582876205444336, 2.9448695182800293, 11.42699909210205, 15.407469749450684, 0.5378808379173279], step: 193400, lr: 3.1625799731458644e-05, reference_loss: 32.37550735473633 2023-09-05 16:59:38,722 44k INFO ====> Epoch: 9210, cost 14.43 s 2023-09-05 16:59:53,418 44k INFO ====> Epoch: 9211, cost 14.70 s 2023-09-05 17:00:08,102 44k INFO ====> Epoch: 9212, cost 14.68 s 2023-09-05 17:00:22,712 44k INFO ====> Epoch: 9213, cost 14.61 s 2023-09-05 17:00:37,141 44k INFO ====> Epoch: 9214, cost 14.43 s 2023-09-05 17:00:51,666 44k INFO ====> Epoch: 9215, cost 14.53 s 2023-09-05 17:01:06,230 44k INFO ====> Epoch: 9216, cost 14.56 s 2023-09-05 17:01:20,704 44k INFO ====> Epoch: 9217, cost 14.47 s 2023-09-05 17:01:35,010 44k INFO ====> Epoch: 9218, cost 14.31 s 2023-09-05 17:01:49,373 44k INFO ====> Epoch: 9219, cost 14.36 s 2023-09-05 17:01:50,135 44k INFO Train Epoch: 9220 [0%] 2023-09-05 17:01:50,136 44k INFO Losses: [2.2377207279205322, 2.874527931213379, 8.536550521850586, 14.892508506774902, 0.6124133467674255], step: 193600, lr: 3.158628971127407e-05, reference_loss: 29.15372085571289 2023-09-05 17:01:55,581 44k INFO Saving model and optimizer state at iteration 9220 to ./logs\44k\G_193600.pth 2023-09-05 17:01:56,358 44k INFO Saving model and optimizer state at iteration 9220 to ./logs\44k\D_193600.pth 2023-09-05 17:01:57,478 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_191200.pth 2023-09-05 17:01:57,511 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_191200.pth 2023-09-05 17:02:11,437 44k INFO ====> Epoch: 9220, cost 22.06 s 2023-09-05 17:02:25,866 44k INFO ====> Epoch: 9221, cost 14.43 s 2023-09-05 17:02:40,228 44k INFO ====> Epoch: 9222, cost 14.36 s 2023-09-05 17:02:54,833 44k INFO ====> Epoch: 9223, cost 14.61 s 2023-09-05 17:03:09,309 44k INFO ====> Epoch: 9224, cost 14.48 s 2023-09-05 17:03:23,757 44k INFO ====> Epoch: 9225, cost 14.45 s 2023-09-05 17:03:38,164 44k INFO ====> Epoch: 9226, cost 14.41 s 2023-09-05 17:03:52,739 44k INFO ====> Epoch: 9227, cost 14.57 s 2023-09-05 17:04:07,403 44k INFO ====> Epoch: 9228, cost 14.66 s 2023-09-05 17:04:15,969 44k INFO Train Epoch: 9229 [52%] 2023-09-05 17:04:15,969 44k INFO Losses: [2.4368929862976074, 2.395195960998535, 8.641945838928223, 15.582754135131836, 0.6303460001945496], step: 193800, lr: 3.1550772897455687e-05, reference_loss: 29.687135696411133 2023-09-05 17:04:22,279 44k INFO ====> Epoch: 9229, cost 14.88 s 2023-09-05 17:04:36,612 44k INFO ====> Epoch: 9230, cost 14.33 s 2023-09-05 17:04:51,143 44k INFO ====> Epoch: 9231, cost 14.53 s 2023-09-05 17:05:05,677 44k INFO ====> Epoch: 9232, cost 14.53 s 2023-09-05 17:05:20,207 44k INFO ====> Epoch: 9233, cost 14.53 s 2023-09-05 17:05:34,588 44k INFO ====> Epoch: 9234, cost 14.38 s 2023-09-05 17:05:49,324 44k INFO ====> Epoch: 9235, cost 14.74 s 2023-09-05 17:06:04,150 44k INFO ====> Epoch: 9236, cost 14.83 s 2023-09-05 17:06:18,690 44k INFO ====> Epoch: 9237, cost 14.54 s 2023-09-05 17:06:33,406 44k INFO ====> Epoch: 9238, cost 14.72 s 2023-09-05 17:06:34,817 44k INFO Train Epoch: 9239 [5%] 2023-09-05 17:06:34,817 44k INFO Losses: [2.309704303741455, 2.668931007385254, 9.94482421875, 15.102500915527344, 0.4989118278026581], step: 194000, lr: 3.151135660807796e-05, reference_loss: 30.524871826171875 2023-09-05 17:06:48,286 44k INFO ====> Epoch: 9239, cost 14.88 s 2023-09-05 17:07:02,946 44k INFO ====> Epoch: 9240, cost 14.66 s 2023-09-05 17:07:17,318 44k INFO ====> Epoch: 9241, cost 14.37 s 2023-09-05 17:07:31,543 44k INFO ====> Epoch: 9242, cost 14.22 s 2023-09-05 17:07:45,980 44k INFO ====> Epoch: 9243, cost 14.44 s 2023-09-05 17:08:00,558 44k INFO ====> Epoch: 9244, cost 14.58 s 2023-09-05 17:08:15,290 44k INFO ====> Epoch: 9245, cost 14.73 s 2023-09-05 17:08:29,696 44k INFO ====> Epoch: 9246, cost 14.41 s 2023-09-05 17:08:44,149 44k INFO ====> Epoch: 9247, cost 14.45 s 2023-09-05 17:08:53,225 44k INFO Train Epoch: 9248 [57%] 2023-09-05 17:08:53,225 44k INFO Losses: [2.266561985015869, 2.6063010692596436, 10.211487770080566, 15.946676254272461, 0.49653923511505127], step: 194200, lr: 3.147592405186309e-05, reference_loss: 31.52756690979004 2023-09-05 17:08:58,842 44k INFO ====> Epoch: 9248, cost 14.69 s 2023-09-05 17:09:13,324 44k INFO ====> Epoch: 9249, cost 14.48 s 2023-09-05 17:09:27,783 44k INFO ====> Epoch: 9250, cost 14.46 s 2023-09-05 17:09:42,103 44k INFO ====> Epoch: 9251, cost 14.32 s 2023-09-05 17:09:56,789 44k INFO ====> Epoch: 9252, cost 14.69 s 2023-09-05 17:10:11,285 44k INFO ====> Epoch: 9253, cost 14.50 s 2023-09-05 17:10:25,555 44k INFO ====> Epoch: 9254, cost 14.27 s 2023-09-05 17:10:39,966 44k INFO ====> Epoch: 9255, cost 14.41 s 2023-09-05 17:10:54,613 44k INFO ====> Epoch: 9256, cost 14.65 s 2023-09-05 17:11:09,400 44k INFO ====> Epoch: 9257, cost 14.79 s 2023-09-05 17:11:11,552 44k INFO Train Epoch: 9258 [10%] 2023-09-05 17:11:11,553 44k INFO Losses: [2.3299198150634766, 2.92279314994812, 9.552420616149902, 14.361861228942871, 0.6798816919326782], step: 194400, lr: 3.1436601270931795e-05, reference_loss: 29.846878051757812 2023-09-05 17:11:17,150 44k INFO Saving model and optimizer state at iteration 9258 to ./logs\44k\G_194400.pth 2023-09-05 17:11:17,743 44k INFO Saving model and optimizer state at iteration 9258 to ./logs\44k\D_194400.pth 2023-09-05 17:11:18,880 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_192000.pth 2023-09-05 17:11:18,915 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_192000.pth 2023-09-05 17:11:31,248 44k INFO ====> Epoch: 9258, cost 21.85 s 2023-09-05 17:11:45,811 44k INFO ====> Epoch: 9259, cost 14.56 s 2023-09-05 17:12:00,434 44k INFO ====> Epoch: 9260, cost 14.62 s 2023-09-05 17:12:15,180 44k INFO ====> Epoch: 9261, cost 14.75 s 2023-09-05 17:12:29,935 44k INFO ====> Epoch: 9262, cost 14.76 s 2023-09-05 17:12:44,545 44k INFO ====> Epoch: 9263, cost 14.61 s 2023-09-05 17:12:59,047 44k INFO ====> Epoch: 9264, cost 14.50 s 2023-09-05 17:13:13,462 44k INFO ====> Epoch: 9265, cost 14.41 s 2023-09-05 17:13:27,798 44k INFO ====> Epoch: 9266, cost 14.34 s 2023-09-05 17:13:37,646 44k INFO Train Epoch: 9267 [62%] 2023-09-05 17:13:37,647 44k INFO Losses: [2.0772926807403564, 2.9233973026275635, 9.703357696533203, 15.712907791137695, 0.46552515029907227], step: 194600, lr: 3.1401252772433606e-05, reference_loss: 30.88248062133789 2023-09-05 17:13:42,657 44k INFO ====> Epoch: 9267, cost 14.86 s 2023-09-05 17:13:57,277 44k INFO ====> Epoch: 9268, cost 14.62 s 2023-09-05 17:14:11,910 44k INFO ====> Epoch: 9269, cost 14.63 s 2023-09-05 17:14:26,324 44k INFO ====> Epoch: 9270, cost 14.41 s 2023-09-05 17:14:40,818 44k INFO ====> Epoch: 9271, cost 14.49 s 2023-09-05 17:14:55,250 44k INFO ====> Epoch: 9272, cost 14.43 s 2023-09-05 17:15:09,910 44k INFO ====> Epoch: 9273, cost 14.66 s 2023-09-05 17:15:24,327 44k INFO ====> Epoch: 9274, cost 14.42 s 2023-09-05 17:15:38,774 44k INFO ====> Epoch: 9275, cost 14.45 s 2023-09-05 17:15:53,354 44k INFO ====> Epoch: 9276, cost 14.58 s 2023-09-05 17:15:56,179 44k INFO Train Epoch: 9277 [14%] 2023-09-05 17:15:56,180 44k INFO Losses: [1.9981040954589844, 3.081743001937866, 12.021214485168457, 15.792165756225586, 0.5608497262001038], step: 194800, lr: 3.136202327811584e-05, reference_loss: 33.45407485961914 2023-09-05 17:16:08,216 44k INFO ====> Epoch: 9277, cost 14.86 s 2023-09-05 17:16:22,699 44k INFO ====> Epoch: 9278, cost 14.48 s 2023-09-05 17:16:37,191 44k INFO ====> Epoch: 9279, cost 14.49 s 2023-09-05 17:16:51,698 44k INFO ====> Epoch: 9280, cost 14.51 s 2023-09-05 17:17:06,415 44k INFO ====> Epoch: 9281, cost 14.72 s 2023-09-05 17:17:20,668 44k INFO ====> Epoch: 9282, cost 14.25 s 2023-09-05 17:17:34,949 44k INFO ====> Epoch: 9283, cost 14.28 s 2023-09-05 17:17:49,590 44k INFO ====> Epoch: 9284, cost 14.64 s 2023-09-05 17:18:04,165 44k INFO ====> Epoch: 9285, cost 14.57 s 2023-09-05 17:18:14,735 44k INFO Train Epoch: 9286 [67%] 2023-09-05 17:18:14,735 44k INFO Losses: [2.0970256328582764, 3.0194921493530273, 9.220012664794922, 13.792935371398926, 0.47297391295433044], step: 195000, lr: 3.132675863792168e-05, reference_loss: 28.602441787719727 2023-09-05 17:18:18,905 44k INFO ====> Epoch: 9286, cost 14.74 s 2023-09-05 17:18:33,350 44k INFO ====> Epoch: 9287, cost 14.45 s 2023-09-05 17:18:47,881 44k INFO ====> Epoch: 9288, cost 14.53 s 2023-09-05 17:19:02,592 44k INFO ====> Epoch: 9289, cost 14.71 s 2023-09-05 17:19:17,147 44k INFO ====> Epoch: 9290, cost 14.56 s 2023-09-05 17:19:31,507 44k INFO ====> Epoch: 9291, cost 14.36 s 2023-09-05 17:19:45,994 44k INFO ====> Epoch: 9292, cost 14.49 s 2023-09-05 17:20:00,841 44k INFO ====> Epoch: 9293, cost 14.85 s 2023-09-05 17:20:15,461 44k INFO ====> Epoch: 9294, cost 14.62 s 2023-09-05 17:20:30,079 44k INFO ====> Epoch: 9295, cost 14.62 s 2023-09-05 17:20:33,555 44k INFO Train Epoch: 9296 [19%] 2023-09-05 17:20:33,555 44k INFO Losses: [2.304050922393799, 2.6631245613098145, 10.141874313354492, 15.465127944946289, 0.6724960803985596], step: 195200, lr: 3.128762220891083e-05, reference_loss: 31.246673583984375 2023-09-05 17:20:39,132 44k INFO Saving model and optimizer state at iteration 9296 to ./logs\44k\G_195200.pth 2023-09-05 17:20:39,738 44k INFO Saving model and optimizer state at iteration 9296 to ./logs\44k\D_195200.pth 2023-09-05 17:20:41,392 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_192800.pth 2023-09-05 17:20:41,428 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_192800.pth 2023-09-05 17:20:52,531 44k INFO ====> Epoch: 9296, cost 22.45 s 2023-09-05 17:21:07,129 44k INFO ====> Epoch: 9297, cost 14.60 s 2023-09-05 17:21:21,732 44k INFO ====> Epoch: 9298, cost 14.60 s 2023-09-05 17:21:35,864 44k INFO ====> Epoch: 9299, cost 14.13 s 2023-09-05 17:21:50,451 44k INFO ====> Epoch: 9300, cost 14.59 s 2023-09-05 17:22:05,126 44k INFO ====> Epoch: 9301, cost 14.67 s 2023-09-05 17:22:19,537 44k INFO ====> Epoch: 9302, cost 14.41 s 2023-09-05 17:22:33,951 44k INFO ====> Epoch: 9303, cost 14.41 s 2023-09-05 17:22:48,349 44k INFO ====> Epoch: 9304, cost 14.40 s 2023-09-05 17:22:59,819 44k INFO Train Epoch: 9305 [71%] 2023-09-05 17:22:59,820 44k INFO Losses: [2.0409746170043945, 2.8480536937713623, 8.931719779968262, 14.050548553466797, 0.6978452801704407], step: 195400, lr: 3.125244122808113e-05, reference_loss: 28.569141387939453 2023-09-05 17:23:03,333 44k INFO ====> Epoch: 9305, cost 14.98 s 2023-09-05 17:23:17,541 44k INFO ====> Epoch: 9306, cost 14.21 s 2023-09-05 17:23:31,904 44k INFO ====> Epoch: 9307, cost 14.36 s 2023-09-05 17:23:46,536 44k INFO ====> Epoch: 9308, cost 14.63 s 2023-09-05 17:24:01,410 44k INFO ====> Epoch: 9309, cost 14.87 s 2023-09-05 17:24:16,202 44k INFO ====> Epoch: 9310, cost 14.79 s 2023-09-05 17:24:30,842 44k INFO ====> Epoch: 9311, cost 14.64 s 2023-09-05 17:24:45,312 44k INFO ====> Epoch: 9312, cost 14.47 s 2023-09-05 17:24:59,985 44k INFO ====> Epoch: 9313, cost 14.67 s 2023-09-05 17:25:14,628 44k INFO ====> Epoch: 9314, cost 14.64 s 2023-09-05 17:25:18,861 44k INFO Train Epoch: 9315 [24%] 2023-09-05 17:25:18,861 44k INFO Losses: [2.421410322189331, 2.9553070068359375, 8.247920989990234, 14.567977905273438, 0.470546692609787], step: 195600, lr: 3.121339764359557e-05, reference_loss: 28.663162231445312 2023-09-05 17:25:29,252 44k INFO ====> Epoch: 9315, cost 14.62 s 2023-09-05 17:25:43,903 44k INFO ====> Epoch: 9316, cost 14.65 s 2023-09-05 17:25:58,626 44k INFO ====> Epoch: 9317, cost 14.72 s 2023-09-05 17:26:13,077 44k INFO ====> Epoch: 9318, cost 14.45 s 2023-09-05 17:26:27,617 44k INFO ====> Epoch: 9319, cost 14.54 s 2023-09-05 17:26:42,138 44k INFO ====> Epoch: 9320, cost 14.52 s 2023-09-05 17:26:56,625 44k INFO ====> Epoch: 9321, cost 14.49 s 2023-09-05 17:27:11,144 44k INFO ====> Epoch: 9322, cost 14.52 s 2023-09-05 17:27:25,631 44k INFO ====> Epoch: 9323, cost 14.49 s 2023-09-05 17:27:37,505 44k INFO Train Epoch: 9324 [76%] 2023-09-05 17:27:37,506 44k INFO Losses: [1.923397183418274, 3.3152570724487305, 12.126072883605957, 13.730691909790039, 0.4887685477733612], step: 195800, lr: 3.117830012366269e-05, reference_loss: 31.584186553955078 2023-09-05 17:27:40,321 44k INFO ====> Epoch: 9324, cost 14.69 s 2023-09-05 17:27:54,707 44k INFO ====> Epoch: 9325, cost 14.39 s 2023-09-05 17:28:09,220 44k INFO ====> Epoch: 9326, cost 14.51 s 2023-09-05 17:28:23,791 44k INFO ====> Epoch: 9327, cost 14.57 s 2023-09-05 17:28:38,312 44k INFO ====> Epoch: 9328, cost 14.52 s 2023-09-05 17:28:52,967 44k INFO ====> Epoch: 9329, cost 14.66 s 2023-09-05 17:29:07,740 44k INFO ====> Epoch: 9330, cost 14.77 s 2023-09-05 17:29:22,097 44k INFO ====> Epoch: 9331, cost 14.36 s 2023-09-05 17:29:36,492 44k INFO ====> Epoch: 9332, cost 14.40 s 2023-09-05 17:29:51,294 44k INFO ====> Epoch: 9333, cost 14.80 s 2023-09-05 17:29:56,297 44k INFO Train Epoch: 9334 [29%] 2023-09-05 17:29:56,297 44k INFO Losses: [2.203108072280884, 2.8384926319122314, 12.096335411071777, 15.289323806762695, 0.6764450669288635], step: 196000, lr: 3.113934916344456e-05, reference_loss: 33.10370635986328 2023-09-05 17:30:01,871 44k INFO Saving model and optimizer state at iteration 9334 to ./logs\44k\G_196000.pth 2023-09-05 17:30:02,583 44k INFO Saving model and optimizer state at iteration 9334 to ./logs\44k\D_196000.pth 2023-09-05 17:30:03,647 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_193600.pth 2023-09-05 17:30:03,684 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_193600.pth 2023-09-05 17:30:13,343 44k INFO ====> Epoch: 9334, cost 22.05 s 2023-09-05 17:30:27,846 44k INFO ====> Epoch: 9335, cost 14.50 s 2023-09-05 17:30:42,588 44k INFO ====> Epoch: 9336, cost 14.74 s 2023-09-05 17:30:57,272 44k INFO ====> Epoch: 9337, cost 14.68 s 2023-09-05 17:31:11,859 44k INFO ====> Epoch: 9338, cost 14.59 s 2023-09-05 17:31:26,041 44k INFO ====> Epoch: 9339, cost 14.18 s 2023-09-05 17:31:40,434 44k INFO ====> Epoch: 9340, cost 14.39 s 2023-09-05 17:31:55,015 44k INFO ====> Epoch: 9341, cost 14.58 s 2023-09-05 17:32:09,592 44k INFO ====> Epoch: 9342, cost 14.58 s 2023-09-05 17:32:22,234 44k INFO Train Epoch: 9343 [81%] 2023-09-05 17:32:22,235 44k INFO Losses: [2.15171480178833, 2.797429323196411, 11.537270545959473, 16.03739356994629, 0.6486716866493225], step: 196200, lr: 3.110433490641174e-05, reference_loss: 33.172481536865234 2023-09-05 17:32:24,326 44k INFO ====> Epoch: 9343, cost 14.73 s 2023-09-05 17:32:38,678 44k INFO ====> Epoch: 9344, cost 14.35 s 2023-09-05 17:32:53,198 44k INFO ====> Epoch: 9345, cost 14.52 s 2023-09-05 17:33:07,801 44k INFO ====> Epoch: 9346, cost 14.60 s 2023-09-05 17:33:22,218 44k INFO ====> Epoch: 9347, cost 14.42 s 2023-09-05 17:33:36,673 44k INFO ====> Epoch: 9348, cost 14.46 s 2023-09-05 17:33:51,150 44k INFO ====> Epoch: 9349, cost 14.48 s 2023-09-05 17:34:05,744 44k INFO ====> Epoch: 9350, cost 14.59 s 2023-09-05 17:34:20,210 44k INFO ====> Epoch: 9351, cost 14.47 s 2023-09-05 17:34:34,704 44k INFO ====> Epoch: 9352, cost 14.49 s 2023-09-05 17:34:40,363 44k INFO Train Epoch: 9353 [33%] 2023-09-05 17:34:40,363 44k INFO Losses: [2.1495203971862793, 2.7354211807250977, 12.787885665893555, 16.111087799072266, 0.619865894317627], step: 196400, lr: 3.106547635072569e-05, reference_loss: 34.403778076171875 2023-09-05 17:34:49,500 44k INFO ====> Epoch: 9353, cost 14.80 s 2023-09-05 17:35:04,242 44k INFO ====> Epoch: 9354, cost 14.74 s 2023-09-05 17:35:18,665 44k INFO ====> Epoch: 9355, cost 14.42 s 2023-09-05 17:35:33,114 44k INFO ====> Epoch: 9356, cost 14.45 s 2023-09-05 17:35:47,722 44k INFO ====> Epoch: 9357, cost 14.61 s 2023-09-05 17:36:02,490 44k INFO ====> Epoch: 9358, cost 14.77 s 2023-09-05 17:36:16,944 44k INFO ====> Epoch: 9359, cost 14.45 s 2023-09-05 17:36:31,468 44k INFO ====> Epoch: 9360, cost 14.52 s 2023-09-05 17:36:46,022 44k INFO ====> Epoch: 9361, cost 14.55 s 2023-09-05 17:36:59,562 44k INFO Train Epoch: 9362 [86%] 2023-09-05 17:36:59,563 44k INFO Losses: [2.1767382621765137, 3.0113525390625, 10.239583969116211, 14.780370712280273, 0.6347081661224365], step: 196600, lr: 3.103054515906584e-05, reference_loss: 30.842754364013672 2023-09-05 17:37:00,930 44k INFO ====> Epoch: 9362, cost 14.91 s 2023-09-05 17:37:15,512 44k INFO ====> Epoch: 9363, cost 14.58 s 2023-09-05 17:37:29,887 44k INFO ====> Epoch: 9364, cost 14.38 s 2023-09-05 17:37:44,338 44k INFO ====> Epoch: 9365, cost 14.45 s 2023-09-05 17:37:59,023 44k INFO ====> Epoch: 9366, cost 14.69 s 2023-09-05 17:38:13,660 44k INFO ====> Epoch: 9367, cost 14.64 s 2023-09-05 17:38:28,161 44k INFO ====> Epoch: 9368, cost 14.50 s 2023-09-05 17:38:42,765 44k INFO ====> Epoch: 9369, cost 14.60 s 2023-09-05 17:38:57,431 44k INFO ====> Epoch: 9370, cost 14.67 s 2023-09-05 17:39:11,952 44k INFO ====> Epoch: 9371, cost 14.52 s 2023-09-05 17:39:18,256 44k INFO Train Epoch: 9372 [38%] 2023-09-05 17:39:18,256 44k INFO Losses: [2.4133996963500977, 2.509052276611328, 9.445961952209473, 13.068058013916016, 0.596778929233551], step: 196800, lr: 3.099177878869786e-05, reference_loss: 28.03325080871582 2023-09-05 17:39:23,721 44k INFO Saving model and optimizer state at iteration 9372 to ./logs\44k\G_196800.pth 2023-09-05 17:39:24,284 44k INFO Saving model and optimizer state at iteration 9372 to ./logs\44k\D_196800.pth 2023-09-05 17:39:25,535 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_194400.pth 2023-09-05 17:39:25,569 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_194400.pth 2023-09-05 17:39:33,760 44k INFO ====> Epoch: 9372, cost 21.81 s 2023-09-05 17:39:48,403 44k INFO ====> Epoch: 9373, cost 14.64 s 2023-09-05 17:40:03,066 44k INFO ====> Epoch: 9374, cost 14.66 s 2023-09-05 17:40:17,533 44k INFO ====> Epoch: 9375, cost 14.47 s 2023-09-05 17:40:32,029 44k INFO ====> Epoch: 9376, cost 14.50 s 2023-09-05 17:40:46,631 44k INFO ====> Epoch: 9377, cost 14.60 s 2023-09-05 17:41:01,299 44k INFO ====> Epoch: 9378, cost 14.67 s 2023-09-05 17:41:15,715 44k INFO ====> Epoch: 9379, cost 14.42 s 2023-09-05 17:41:30,038 44k INFO ====> Epoch: 9380, cost 14.32 s 2023-09-05 17:41:44,101 44k INFO Train Epoch: 9381 [90%] 2023-09-05 17:41:44,102 44k INFO Losses: [2.096658706665039, 2.884495973587036, 9.34390926361084, 15.462557792663574, 0.7547686696052551], step: 197000, lr: 3.095693046535249e-05, reference_loss: 30.542388916015625 2023-09-05 17:41:44,771 44k INFO ====> Epoch: 9381, cost 14.73 s 2023-09-05 17:41:59,632 44k INFO ====> Epoch: 9382, cost 14.86 s 2023-09-05 17:42:14,334 44k INFO ====> Epoch: 9383, cost 14.70 s 2023-09-05 17:42:28,908 44k INFO ====> Epoch: 9384, cost 14.57 s 2023-09-05 17:42:43,520 44k INFO ====> Epoch: 9385, cost 14.61 s 2023-09-05 17:42:58,083 44k INFO ====> Epoch: 9386, cost 14.56 s 2023-09-05 17:43:12,665 44k INFO ====> Epoch: 9387, cost 14.58 s 2023-09-05 17:43:27,094 44k INFO ====> Epoch: 9388, cost 14.43 s 2023-09-05 17:43:41,476 44k INFO ====> Epoch: 9389, cost 14.38 s 2023-09-05 17:43:56,052 44k INFO ====> Epoch: 9390, cost 14.58 s 2023-09-05 17:44:03,228 44k INFO Train Epoch: 9391 [43%] 2023-09-05 17:44:03,229 44k INFO Losses: [2.1642024517059326, 2.8436150550842285, 9.848041534423828, 14.94177532196045, 0.5426580309867859], step: 197200, lr: 3.091825606160859e-05, reference_loss: 30.34029197692871 2023-09-05 17:44:11,123 44k INFO ====> Epoch: 9391, cost 15.07 s 2023-09-05 17:44:25,486 44k INFO ====> Epoch: 9392, cost 14.36 s 2023-09-05 17:44:39,845 44k INFO ====> Epoch: 9393, cost 14.36 s 2023-09-05 17:44:54,388 44k INFO ====> Epoch: 9394, cost 14.54 s 2023-09-05 17:45:09,014 44k INFO ====> Epoch: 9395, cost 14.63 s 2023-09-05 17:45:23,360 44k INFO ====> Epoch: 9396, cost 14.35 s 2023-09-05 17:45:37,718 44k INFO ====> Epoch: 9397, cost 14.36 s 2023-09-05 17:45:52,315 44k INFO ====> Epoch: 9398, cost 14.60 s 2023-09-05 17:46:06,878 44k INFO ====> Epoch: 9399, cost 14.56 s 2023-09-05 17:46:21,382 44k INFO Train Epoch: 9400 [95%] 2023-09-05 17:46:21,382 44k INFO Losses: [1.929947853088379, 3.0061726570129395, 11.957880973815918, 14.737940788269043, 0.5249302387237549], step: 197400, lr: 3.0883490409986724e-05, reference_loss: 32.1568717956543 2023-09-05 17:46:21,646 44k INFO ====> Epoch: 9400, cost 14.77 s 2023-09-05 17:46:36,195 44k INFO ====> Epoch: 9401, cost 14.55 s 2023-09-05 17:46:50,690 44k INFO ====> Epoch: 9402, cost 14.49 s 2023-09-05 17:47:05,305 44k INFO ====> Epoch: 9403, cost 14.62 s 2023-09-05 17:47:19,744 44k INFO ====> Epoch: 9404, cost 14.44 s 2023-09-05 17:47:34,049 44k INFO ====> Epoch: 9405, cost 14.30 s 2023-09-05 17:47:48,613 44k INFO ====> Epoch: 9406, cost 14.56 s 2023-09-05 17:48:03,452 44k INFO ====> Epoch: 9407, cost 14.84 s 2023-09-05 17:48:18,051 44k INFO ====> Epoch: 9408, cost 14.60 s 2023-09-05 17:48:32,596 44k INFO ====> Epoch: 9409, cost 14.54 s 2023-09-05 17:48:40,373 44k INFO Train Epoch: 9410 [48%] 2023-09-05 17:48:40,374 44k INFO Losses: [2.0446481704711914, 2.901050329208374, 12.08765697479248, 16.133953094482422, 0.6062913537025452], step: 197600, lr: 3.0844907754691685e-05, reference_loss: 33.77360153198242 2023-09-05 17:48:45,890 44k INFO Saving model and optimizer state at iteration 9410 to ./logs\44k\G_197600.pth 2023-09-05 17:48:46,461 44k INFO Saving model and optimizer state at iteration 9410 to ./logs\44k\D_197600.pth 2023-09-05 17:48:47,776 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_195200.pth 2023-09-05 17:48:47,812 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_195200.pth 2023-09-05 17:48:54,684 44k INFO ====> Epoch: 9410, cost 22.09 s 2023-09-05 17:49:09,303 44k INFO ====> Epoch: 9411, cost 14.62 s 2023-09-05 17:49:23,725 44k INFO ====> Epoch: 9412, cost 14.42 s 2023-09-05 17:49:38,052 44k INFO ====> Epoch: 9413, cost 14.33 s 2023-09-05 17:49:52,877 44k INFO ====> Epoch: 9414, cost 14.82 s 2023-09-05 17:50:07,630 44k INFO ====> Epoch: 9415, cost 14.75 s 2023-09-05 17:50:22,190 44k INFO ====> Epoch: 9416, cost 14.56 s 2023-09-05 17:50:36,695 44k INFO ====> Epoch: 9417, cost 14.51 s 2023-09-05 17:50:51,094 44k INFO ====> Epoch: 9418, cost 14.40 s 2023-09-05 17:51:05,648 44k INFO ====> Epoch: 9419, cost 14.55 s 2023-09-05 17:51:06,336 44k INFO Train Epoch: 9420 [0%] 2023-09-05 17:51:06,336 44k INFO Losses: [2.2185935974121094, 2.885871171951294, 10.194805145263672, 14.860767364501953, 0.5386750102043152], step: 197800, lr: 3.080637330059638e-05, reference_loss: 30.698713302612305 2023-09-05 17:51:20,360 44k INFO ====> Epoch: 9420, cost 14.71 s 2023-09-05 17:51:34,657 44k INFO ====> Epoch: 9421, cost 14.30 s 2023-09-05 17:51:49,178 44k INFO ====> Epoch: 9422, cost 14.52 s 2023-09-05 17:52:03,867 44k INFO ====> Epoch: 9423, cost 14.69 s 2023-09-05 17:52:18,371 44k INFO ====> Epoch: 9424, cost 14.50 s 2023-09-05 17:52:32,824 44k INFO ====> Epoch: 9425, cost 14.45 s 2023-09-05 17:52:47,443 44k INFO ====> Epoch: 9426, cost 14.62 s 2023-09-05 17:53:01,994 44k INFO ====> Epoch: 9427, cost 14.55 s 2023-09-05 17:53:16,609 44k INFO ====> Epoch: 9428, cost 14.61 s 2023-09-05 17:53:25,060 44k INFO Train Epoch: 9429 [52%] 2023-09-05 17:53:25,061 44k INFO Losses: [2.1783854961395264, 2.737276554107666, 8.763815879821777, 15.81457805633545, 0.7033383846282959], step: 198000, lr: 3.077173345416495e-05, reference_loss: 30.19739532470703 2023-09-05 17:53:31,297 44k INFO ====> Epoch: 9429, cost 14.69 s 2023-09-05 17:53:45,795 44k INFO ====> Epoch: 9430, cost 14.50 s 2023-09-05 17:54:00,605 44k INFO ====> Epoch: 9431, cost 14.81 s 2023-09-05 17:54:15,097 44k INFO ====> Epoch: 9432, cost 14.49 s 2023-09-05 17:54:29,660 44k INFO ====> Epoch: 9433, cost 14.56 s 2023-09-05 17:54:44,203 44k INFO ====> Epoch: 9434, cost 14.54 s 2023-09-05 17:54:59,041 44k INFO ====> Epoch: 9435, cost 14.84 s 2023-09-05 17:55:13,571 44k INFO ====> Epoch: 9436, cost 14.53 s 2023-09-05 17:55:27,927 44k INFO ====> Epoch: 9437, cost 14.36 s 2023-09-05 17:55:42,216 44k INFO ====> Epoch: 9438, cost 14.29 s 2023-09-05 17:55:43,647 44k INFO Train Epoch: 9439 [5%] 2023-09-05 17:55:43,647 44k INFO Losses: [1.988966941833496, 3.1581764221191406, 10.487067222595215, 15.934578895568848, 0.5138458013534546], step: 198200, lr: 3.073329041651176e-05, reference_loss: 32.082637786865234 2023-09-05 17:55:57,100 44k INFO ====> Epoch: 9439, cost 14.88 s 2023-09-05 17:56:11,650 44k INFO ====> Epoch: 9440, cost 14.55 s 2023-09-05 17:56:26,099 44k INFO ====> Epoch: 9441, cost 14.45 s 2023-09-05 17:56:40,574 44k INFO ====> Epoch: 9442, cost 14.48 s 2023-09-05 17:56:55,114 44k INFO ====> Epoch: 9443, cost 14.54 s 2023-09-05 17:57:09,705 44k INFO ====> Epoch: 9444, cost 14.59 s 2023-09-05 17:57:24,095 44k INFO ====> Epoch: 9445, cost 14.39 s 2023-09-05 17:57:38,479 44k INFO ====> Epoch: 9446, cost 14.38 s 2023-09-05 17:57:53,184 44k INFO ====> Epoch: 9447, cost 14.71 s 2023-09-05 17:58:02,526 44k INFO Train Epoch: 9448 [57%] 2023-09-05 17:58:02,526 44k INFO Losses: [2.3089590072631836, 2.5278091430664062, 8.863168716430664, 14.410378456115723, 0.729712724685669], step: 198400, lr: 3.06987327472278e-05, reference_loss: 28.840028762817383 2023-09-05 17:58:08,038 44k INFO Saving model and optimizer state at iteration 9448 to ./logs\44k\G_198400.pth 2023-09-05 17:58:08,647 44k INFO Saving model and optimizer state at iteration 9448 to ./logs\44k\D_198400.pth 2023-09-05 17:58:09,885 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_196000.pth 2023-09-05 17:58:09,924 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_196000.pth 2023-09-05 17:58:15,327 44k INFO ====> Epoch: 9448, cost 22.14 s 2023-09-05 17:58:29,813 44k INFO ====> Epoch: 9449, cost 14.49 s 2023-09-05 17:58:44,733 44k INFO ====> Epoch: 9450, cost 14.92 s 2023-09-05 17:58:59,484 44k INFO ====> Epoch: 9451, cost 14.75 s 2023-09-05 17:59:14,137 44k INFO ====> Epoch: 9452, cost 14.65 s 2023-09-05 17:59:28,465 44k INFO ====> Epoch: 9453, cost 14.33 s 2023-09-05 17:59:42,989 44k INFO ====> Epoch: 9454, cost 14.53 s 2023-09-05 17:59:57,698 44k INFO ====> Epoch: 9455, cost 14.71 s 2023-09-05 18:00:12,417 44k INFO ====> Epoch: 9456, cost 14.72 s 2023-09-05 18:00:26,800 44k INFO ====> Epoch: 9457, cost 14.38 s 2023-09-05 18:00:28,915 44k INFO Train Epoch: 9458 [10%] 2023-09-05 18:00:28,915 44k INFO Losses: [1.9672493934631348, 3.218616247177124, 9.836297988891602, 14.218073844909668, 0.6337855458259583], step: 198600, lr: 3.0660380909146775e-05, reference_loss: 29.8740234375 2023-09-05 18:00:41,532 44k INFO ====> Epoch: 9458, cost 14.73 s 2023-09-05 18:00:56,234 44k INFO ====> Epoch: 9459, cost 14.70 s 2023-09-05 18:01:10,835 44k INFO ====> Epoch: 9460, cost 14.60 s 2023-09-05 18:01:25,169 44k INFO ====> Epoch: 9461, cost 14.33 s 2023-09-05 18:01:39,546 44k INFO ====> Epoch: 9462, cost 14.38 s 2023-09-05 18:01:54,215 44k INFO ====> Epoch: 9463, cost 14.67 s 2023-09-05 18:02:08,753 44k INFO ====> Epoch: 9464, cost 14.54 s 2023-09-05 18:02:23,233 44k INFO ====> Epoch: 9465, cost 14.48 s 2023-09-05 18:02:37,807 44k INFO ====> Epoch: 9466, cost 14.57 s 2023-09-05 18:02:47,687 44k INFO Train Epoch: 9467 [62%] 2023-09-05 18:02:47,688 44k INFO Losses: [2.174039840698242, 2.601710319519043, 9.85824203491211, 16.550676345825195, 0.5115352869033813], step: 198800, lr: 3.0625905222058965e-05, reference_loss: 31.696205139160156 2023-09-05 18:02:52,610 44k INFO ====> Epoch: 9467, cost 14.80 s 2023-09-05 18:03:07,207 44k INFO ====> Epoch: 9468, cost 14.60 s 2023-09-05 18:03:21,641 44k INFO ====> Epoch: 9469, cost 14.43 s 2023-09-05 18:03:36,262 44k INFO ====> Epoch: 9470, cost 14.62 s 2023-09-05 18:03:50,708 44k INFO ====> Epoch: 9471, cost 14.45 s 2023-09-05 18:04:05,471 44k INFO ====> Epoch: 9472, cost 14.76 s 2023-09-05 18:04:19,996 44k INFO ====> Epoch: 9473, cost 14.52 s 2023-09-05 18:04:34,527 44k INFO ====> Epoch: 9474, cost 14.53 s 2023-09-05 18:04:49,034 44k INFO ====> Epoch: 9475, cost 14.51 s 2023-09-05 18:05:03,675 44k INFO ====> Epoch: 9476, cost 14.64 s 2023-09-05 18:05:06,475 44k INFO Train Epoch: 9477 [14%] 2023-09-05 18:05:06,475 44k INFO Losses: [2.2355856895446777, 2.901779890060425, 10.332679748535156, 15.474679946899414, 0.5781113505363464], step: 199000, lr: 3.05876443671946e-05, reference_loss: 31.522836685180664 2023-09-05 18:05:18,450 44k INFO ====> Epoch: 9477, cost 14.78 s 2023-09-05 18:05:32,806 44k INFO ====> Epoch: 9478, cost 14.36 s 2023-09-05 18:05:47,245 44k INFO ====> Epoch: 9479, cost 14.44 s 2023-09-05 18:06:02,258 44k INFO ====> Epoch: 9480, cost 15.01 s 2023-09-05 18:06:16,962 44k INFO ====> Epoch: 9481, cost 14.70 s 2023-09-05 18:06:31,399 44k INFO ====> Epoch: 9482, cost 14.44 s 2023-09-05 18:06:45,946 44k INFO ====> Epoch: 9483, cost 14.55 s 2023-09-05 18:07:00,514 44k INFO ====> Epoch: 9484, cost 14.57 s 2023-09-05 18:07:15,098 44k INFO ====> Epoch: 9485, cost 14.58 s 2023-09-05 18:07:25,681 44k INFO Train Epoch: 9486 [67%] 2023-09-05 18:07:25,681 44k INFO Losses: [2.2054073810577393, 2.716642379760742, 9.569421768188477, 16.13672637939453, 0.6163732409477234], step: 199200, lr: 3.0553250467814116e-05, reference_loss: 31.244569778442383 2023-09-05 18:07:31,145 44k INFO Saving model and optimizer state at iteration 9486 to ./logs\44k\G_199200.pth 2023-09-05 18:07:31,825 44k INFO Saving model and optimizer state at iteration 9486 to ./logs\44k\D_199200.pth 2023-09-05 18:07:32,902 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_196800.pth 2023-09-05 18:07:32,941 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_196800.pth 2023-09-05 18:07:36,873 44k INFO ====> Epoch: 9486, cost 21.78 s 2023-09-05 18:07:51,438 44k INFO ====> Epoch: 9487, cost 14.56 s 2023-09-05 18:08:06,154 44k INFO ====> Epoch: 9488, cost 14.72 s 2023-09-05 18:08:20,687 44k INFO ====> Epoch: 9489, cost 14.53 s 2023-09-05 18:08:35,113 44k INFO ====> Epoch: 9490, cost 14.43 s 2023-09-05 18:08:49,665 44k INFO ====> Epoch: 9491, cost 14.55 s 2023-09-05 18:09:04,372 44k INFO ====> Epoch: 9492, cost 14.71 s 2023-09-05 18:09:18,929 44k INFO ====> Epoch: 9493, cost 14.56 s 2023-09-05 18:09:33,274 44k INFO ====> Epoch: 9494, cost 14.35 s 2023-09-05 18:09:47,737 44k INFO ====> Epoch: 9495, cost 14.46 s 2023-09-05 18:09:51,290 44k INFO Train Epoch: 9496 [19%] 2023-09-05 18:09:51,291 44k INFO Losses: [2.198094367980957, 2.784926176071167, 8.330574989318848, 15.402557373046875, 0.6389546394348145], step: 199400, lr: 3.051508038032421e-05, reference_loss: 29.355106353759766 2023-09-05 18:10:02,556 44k INFO ====> Epoch: 9496, cost 14.82 s 2023-09-05 18:10:17,125 44k INFO ====> Epoch: 9497, cost 14.57 s 2023-09-05 18:10:31,670 44k INFO ====> Epoch: 9498, cost 14.55 s 2023-09-05 18:10:46,263 44k INFO ====> Epoch: 9499, cost 14.59 s 2023-09-05 18:11:00,937 44k INFO ====> Epoch: 9500, cost 14.67 s 2023-09-05 18:11:15,462 44k INFO ====> Epoch: 9501, cost 14.53 s 2023-09-05 18:11:29,921 44k INFO ====> Epoch: 9502, cost 14.46 s 2023-09-05 18:11:44,465 44k INFO ====> Epoch: 9503, cost 14.54 s 2023-09-05 18:11:59,366 44k INFO ====> Epoch: 9504, cost 14.90 s 2023-09-05 18:12:11,078 44k INFO Train Epoch: 9505 [71%] 2023-09-05 18:12:11,078 44k INFO Losses: [2.4343483448028564, 2.515075206756592, 8.495670318603516, 14.262187957763672, 0.5739321112632751], step: 199600, lr: 3.048076807462361e-05, reference_loss: 28.28121566772461 2023-09-05 18:12:14,608 44k INFO ====> Epoch: 9505, cost 15.24 s 2023-09-05 18:12:29,111 44k INFO ====> Epoch: 9506, cost 14.50 s 2023-09-05 18:12:43,644 44k INFO ====> Epoch: 9507, cost 14.53 s 2023-09-05 18:12:58,203 44k INFO ====> Epoch: 9508, cost 14.56 s 2023-09-05 18:13:12,749 44k INFO ====> Epoch: 9509, cost 14.55 s 2023-09-05 18:13:27,623 44k INFO ====> Epoch: 9510, cost 14.87 s 2023-09-05 18:13:43,007 44k INFO ====> Epoch: 9511, cost 15.38 s 2023-09-05 18:13:57,672 44k INFO ====> Epoch: 9512, cost 14.67 s 2023-09-05 18:14:12,320 44k INFO ====> Epoch: 9513, cost 14.65 s 2023-09-05 18:14:26,928 44k INFO ====> Epoch: 9514, cost 14.61 s 2023-09-05 18:14:31,099 44k INFO Train Epoch: 9515 [24%] 2023-09-05 18:14:31,099 44k INFO Losses: [2.4948835372924805, 2.24006724357605, 9.998172760009766, 13.712753295898438, 0.5620784759521484], step: 199800, lr: 3.0442688539178e-05, reference_loss: 29.00795555114746 2023-09-05 18:14:41,717 44k INFO ====> Epoch: 9515, cost 14.79 s 2023-09-05 18:14:56,121 44k INFO ====> Epoch: 9516, cost 14.40 s 2023-09-05 18:15:10,750 44k INFO ====> Epoch: 9517, cost 14.63 s 2023-09-05 18:15:25,352 44k INFO ====> Epoch: 9518, cost 14.60 s 2023-09-05 18:15:40,058 44k INFO ====> Epoch: 9519, cost 14.71 s 2023-09-05 18:15:54,823 44k INFO ====> Epoch: 9520, cost 14.77 s 2023-09-05 18:16:09,664 44k INFO ====> Epoch: 9521, cost 14.84 s 2023-09-05 18:16:24,358 44k INFO ====> Epoch: 9522, cost 14.69 s 2023-09-05 18:16:38,986 44k INFO ====> Epoch: 9523, cost 14.63 s 2023-09-05 18:16:51,135 44k INFO Train Epoch: 9524 [76%] 2023-09-05 18:16:51,136 44k INFO Losses: [1.9974210262298584, 2.7362313270568848, 12.379904747009277, 17.263992309570312, 0.5909762382507324], step: 200000, lr: 3.0408457633590146e-05, reference_loss: 34.96852493286133 2023-09-05 18:16:56,780 44k INFO Saving model and optimizer state at iteration 9524 to ./logs\44k\G_200000.pth 2023-09-05 18:16:57,423 44k INFO Saving model and optimizer state at iteration 9524 to ./logs\44k\D_200000.pth 2023-09-05 18:16:58,691 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_197600.pth 2023-09-05 18:16:58,728 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_197600.pth 2023-09-05 18:17:01,297 44k INFO ====> Epoch: 9524, cost 22.31 s 2023-09-05 18:17:16,000 44k INFO ====> Epoch: 9525, cost 14.70 s 2023-09-05 18:17:30,510 44k INFO ====> Epoch: 9526, cost 14.51 s 2023-09-05 18:17:45,002 44k INFO ====> Epoch: 9527, cost 14.49 s 2023-09-05 18:17:59,732 44k INFO ====> Epoch: 9528, cost 14.73 s 2023-09-05 18:18:14,572 44k INFO ====> Epoch: 9529, cost 14.84 s 2023-09-05 18:18:28,978 44k INFO ====> Epoch: 9530, cost 14.41 s 2023-09-05 18:18:43,518 44k INFO ====> Epoch: 9531, cost 14.54 s 2023-09-05 18:18:58,143 44k INFO ====> Epoch: 9532, cost 14.62 s 2023-09-05 18:19:12,743 44k INFO ====> Epoch: 9533, cost 14.60 s 2023-09-05 18:19:17,592 44k INFO Train Epoch: 9534 [29%] 2023-09-05 18:19:17,592 44k INFO Losses: [2.173975706100464, 2.729224443435669, 9.723862648010254, 14.408199310302734, 0.5805919170379639], step: 200200, lr: 3.03704684353695e-05, reference_loss: 29.615854263305664 2023-09-05 18:19:27,248 44k INFO ====> Epoch: 9534, cost 14.51 s 2023-09-05 18:19:41,516 44k INFO ====> Epoch: 9535, cost 14.27 s 2023-09-05 18:19:56,313 44k INFO ====> Epoch: 9536, cost 14.80 s 2023-09-05 18:20:10,905 44k INFO ====> Epoch: 9537, cost 14.59 s 2023-09-05 18:20:25,477 44k INFO ====> Epoch: 9538, cost 14.57 s 2023-09-05 18:20:39,856 44k INFO ====> Epoch: 9539, cost 14.38 s 2023-09-05 18:20:54,261 44k INFO ====> Epoch: 9540, cost 14.41 s 2023-09-05 18:21:08,978 44k INFO ====> Epoch: 9541, cost 14.72 s 2023-09-05 18:21:23,317 44k INFO ====> Epoch: 9542, cost 14.34 s 2023-09-05 18:21:35,916 44k INFO Train Epoch: 9543 [81%] 2023-09-05 18:21:35,916 44k INFO Losses: [2.278930902481079, 2.48950457572937, 10.668062210083008, 14.565418243408203, 0.5781089067459106], step: 200400, lr: 3.0336318736786475e-05, reference_loss: 30.58002471923828 2023-09-05 18:21:38,021 44k INFO ====> Epoch: 9543, cost 14.70 s 2023-09-05 18:21:52,694 44k INFO ====> Epoch: 9544, cost 14.67 s 2023-09-05 18:22:07,249 44k INFO ====> Epoch: 9545, cost 14.55 s 2023-09-05 18:22:21,905 44k INFO ====> Epoch: 9546, cost 14.66 s 2023-09-05 18:22:36,322 44k INFO ====> Epoch: 9547, cost 14.42 s 2023-09-05 18:22:51,012 44k INFO ====> Epoch: 9548, cost 14.69 s 2023-09-05 18:23:05,827 44k INFO ====> Epoch: 9549, cost 14.82 s 2023-09-05 18:23:20,343 44k INFO ====> Epoch: 9550, cost 14.52 s 2023-09-05 18:23:34,845 44k INFO ====> Epoch: 9551, cost 14.50 s 2023-09-05 18:23:49,537 44k INFO ====> Epoch: 9552, cost 14.69 s 2023-09-05 18:23:55,136 44k INFO Train Epoch: 9553 [33%] 2023-09-05 18:23:55,136 44k INFO Losses: [1.9467518329620361, 2.8103668689727783, 10.496167182922363, 15.9248046875, 0.532568633556366], step: 200600, lr: 3.0298419661481073e-05, reference_loss: 31.71065902709961 2023-09-05 18:24:04,603 44k INFO ====> Epoch: 9553, cost 15.07 s 2023-09-05 18:24:19,260 44k INFO ====> Epoch: 9554, cost 14.66 s 2023-09-05 18:24:33,764 44k INFO ====> Epoch: 9555, cost 14.50 s 2023-09-05 18:24:48,290 44k INFO ====> Epoch: 9556, cost 14.53 s 2023-09-05 18:25:02,802 44k INFO ====> Epoch: 9557, cost 14.51 s 2023-09-05 18:25:17,298 44k INFO ====> Epoch: 9558, cost 14.50 s 2023-09-05 18:25:31,646 44k INFO ====> Epoch: 9559, cost 14.35 s 2023-09-05 18:25:46,027 44k INFO ====> Epoch: 9560, cost 14.38 s 2023-09-05 18:26:00,674 44k INFO ====> Epoch: 9561, cost 14.65 s 2023-09-05 18:26:14,169 44k INFO Train Epoch: 9562 [86%] 2023-09-05 18:26:14,169 44k INFO Losses: [2.0665736198425293, 2.8265228271484375, 12.215249061584473, 14.883408546447754, 0.6277934908866882], step: 200800, lr: 3.026435097725305e-05, reference_loss: 32.619544982910156 2023-09-05 18:26:19,734 44k INFO Saving model and optimizer state at iteration 9562 to ./logs\44k\G_200800.pth 2023-09-05 18:26:20,331 44k INFO Saving model and optimizer state at iteration 9562 to ./logs\44k\D_200800.pth 2023-09-05 18:26:21,482 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_198400.pth 2023-09-05 18:26:21,516 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_198400.pth 2023-09-05 18:26:22,652 44k INFO ====> Epoch: 9562, cost 21.98 s 2023-09-05 18:26:37,064 44k INFO ====> Epoch: 9563, cost 14.41 s 2023-09-05 18:26:51,580 44k INFO ====> Epoch: 9564, cost 14.52 s 2023-09-05 18:27:06,158 44k INFO ====> Epoch: 9565, cost 14.58 s 2023-09-05 18:27:20,722 44k INFO ====> Epoch: 9566, cost 14.56 s 2023-09-05 18:27:35,202 44k INFO ====> Epoch: 9567, cost 14.48 s 2023-09-05 18:27:49,703 44k INFO ====> Epoch: 9568, cost 14.50 s 2023-09-05 18:28:04,325 44k INFO ====> Epoch: 9569, cost 14.62 s 2023-09-05 18:28:18,780 44k INFO ====> Epoch: 9570, cost 14.46 s 2023-09-05 18:28:33,086 44k INFO ====> Epoch: 9571, cost 14.31 s 2023-09-05 18:28:39,399 44k INFO Train Epoch: 9572 [38%] 2023-09-05 18:28:39,399 44k INFO Losses: [2.168855905532837, 2.598877191543579, 10.550341606140137, 15.290034294128418, 0.5542401671409607], step: 201000, lr: 3.0226541811061594e-05, reference_loss: 31.1623477935791 2023-09-05 18:28:47,899 44k INFO ====> Epoch: 9572, cost 14.81 s 2023-09-05 18:29:02,402 44k INFO ====> Epoch: 9573, cost 14.50 s 2023-09-05 18:29:16,849 44k INFO ====> Epoch: 9574, cost 14.45 s 2023-09-05 18:29:31,138 44k INFO ====> Epoch: 9575, cost 14.29 s 2023-09-05 18:29:45,656 44k INFO ====> Epoch: 9576, cost 14.52 s 2023-09-05 18:30:00,466 44k INFO ====> Epoch: 9577, cost 14.81 s 2023-09-05 18:30:15,124 44k INFO ====> Epoch: 9578, cost 14.66 s 2023-09-05 18:30:29,592 44k INFO ====> Epoch: 9579, cost 14.47 s 2023-09-05 18:30:44,124 44k INFO ====> Epoch: 9580, cost 14.53 s 2023-09-05 18:30:58,150 44k INFO Train Epoch: 9581 [90%] 2023-09-05 18:30:58,150 44k INFO Losses: [2.2059152126312256, 2.670180320739746, 7.774200916290283, 14.795567512512207, 0.46526211500167847], step: 201200, lr: 3.0192553948995796e-05, reference_loss: 27.9111270904541 2023-09-05 18:30:58,844 44k INFO ====> Epoch: 9581, cost 14.72 s 2023-09-05 18:31:13,500 44k INFO ====> Epoch: 9582, cost 14.66 s 2023-09-05 18:31:27,954 44k INFO ====> Epoch: 9583, cost 14.45 s 2023-09-05 18:31:42,286 44k INFO ====> Epoch: 9584, cost 14.33 s 2023-09-05 18:31:57,110 44k INFO ====> Epoch: 9585, cost 14.82 s 2023-09-05 18:32:11,741 44k INFO ====> Epoch: 9586, cost 14.63 s 2023-09-05 18:32:26,233 44k INFO ====> Epoch: 9587, cost 14.49 s 2023-09-05 18:32:40,681 44k INFO ====> Epoch: 9588, cost 14.45 s 2023-09-05 18:32:55,252 44k INFO ====> Epoch: 9589, cost 14.57 s 2023-09-05 18:33:09,817 44k INFO ====> Epoch: 9590, cost 14.57 s 2023-09-05 18:33:16,857 44k INFO Train Epoch: 9591 [43%] 2023-09-05 18:33:16,857 44k INFO Losses: [2.138482093811035, 2.663318157196045, 9.93972110748291, 16.106882095336914, 0.5008770227432251], step: 201400, lr: 3.0154834478624198e-05, reference_loss: 31.349281311035156 2023-09-05 18:33:24,524 44k INFO ====> Epoch: 9591, cost 14.71 s 2023-09-05 18:33:38,984 44k INFO ====> Epoch: 9592, cost 14.46 s 2023-09-05 18:33:53,526 44k INFO ====> Epoch: 9593, cost 14.54 s 2023-09-05 18:34:08,180 44k INFO ====> Epoch: 9594, cost 14.65 s 2023-09-05 18:34:22,652 44k INFO ====> Epoch: 9595, cost 14.47 s 2023-09-05 18:34:37,082 44k INFO ====> Epoch: 9596, cost 14.43 s 2023-09-05 18:34:51,715 44k INFO ====> Epoch: 9597, cost 14.63 s 2023-09-05 18:35:06,325 44k INFO ====> Epoch: 9598, cost 14.61 s 2023-09-05 18:35:20,649 44k INFO ====> Epoch: 9599, cost 14.32 s 2023-09-05 18:35:35,044 44k INFO Train Epoch: 9600 [95%] 2023-09-05 18:35:35,045 44k INFO Losses: [2.1051676273345947, 2.5092945098876953, 10.529045104980469, 15.31148910522461, 0.2954060733318329], step: 201600, lr: 3.012092724698378e-05, reference_loss: 30.750402450561523 2023-09-05 18:35:40,575 44k INFO Saving model and optimizer state at iteration 9600 to ./logs\44k\G_201600.pth 2023-09-05 18:35:41,354 44k INFO Saving model and optimizer state at iteration 9600 to ./logs\44k\D_201600.pth 2023-09-05 18:35:42,997 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_199200.pth 2023-09-05 18:35:43,034 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_199200.pth 2023-09-05 18:35:43,034 44k INFO ====> Epoch: 9600, cost 22.39 s 2023-09-05 18:35:57,842 44k INFO ====> Epoch: 9601, cost 14.81 s 2023-09-05 18:36:12,548 44k INFO ====> Epoch: 9602, cost 14.71 s 2023-09-05 18:36:27,086 44k INFO ====> Epoch: 9603, cost 14.54 s 2023-09-05 18:36:41,579 44k INFO ====> Epoch: 9604, cost 14.49 s 2023-09-05 18:36:56,133 44k INFO ====> Epoch: 9605, cost 14.55 s 2023-09-05 18:37:10,843 44k INFO ====> Epoch: 9606, cost 14.71 s 2023-09-05 18:37:25,210 44k INFO ====> Epoch: 9607, cost 14.37 s 2023-09-05 18:37:39,581 44k INFO ====> Epoch: 9608, cost 14.37 s 2023-09-05 18:37:54,207 44k INFO ====> Epoch: 9609, cost 14.63 s 2023-09-05 18:38:02,129 44k INFO Train Epoch: 9610 [48%] 2023-09-05 18:38:02,129 44k INFO Losses: [2.044095277786255, 2.9809019565582275, 11.778215408325195, 16.027379989624023, 0.6913384199142456], step: 201800, lr: 3.0083297259643956e-05, reference_loss: 33.52193069458008 2023-09-05 18:38:09,229 44k INFO ====> Epoch: 9610, cost 15.02 s 2023-09-05 18:38:23,701 44k INFO ====> Epoch: 9611, cost 14.47 s 2023-09-05 18:38:38,137 44k INFO ====> Epoch: 9612, cost 14.44 s 2023-09-05 18:38:52,681 44k INFO ====> Epoch: 9613, cost 14.54 s 2023-09-05 18:39:07,242 44k INFO ====> Epoch: 9614, cost 14.56 s 2023-09-05 18:39:21,631 44k INFO ====> Epoch: 9615, cost 14.39 s 2023-09-05 18:39:35,974 44k INFO ====> Epoch: 9616, cost 14.34 s 2023-09-05 18:39:50,514 44k INFO ====> Epoch: 9617, cost 14.54 s 2023-09-05 18:40:05,184 44k INFO ====> Epoch: 9618, cost 14.67 s 2023-09-05 18:40:19,578 44k INFO ====> Epoch: 9619, cost 14.39 s 2023-09-05 18:40:20,281 44k INFO Train Epoch: 9620 [0%] 2023-09-05 18:40:20,282 44k INFO Losses: [2.3629236221313477, 2.769327163696289, 8.96098518371582, 13.637085914611816, 0.5695971846580505], step: 202000, lr: 3.0045714283338546e-05, reference_loss: 28.29991912841797 2023-09-05 18:40:34,223 44k INFO ====> Epoch: 9620, cost 14.64 s 2023-09-05 18:40:48,836 44k INFO ====> Epoch: 9621, cost 14.61 s 2023-09-05 18:41:03,569 44k INFO ====> Epoch: 9622, cost 14.73 s 2023-09-05 18:41:17,996 44k INFO ====> Epoch: 9623, cost 14.43 s 2023-09-05 18:41:32,455 44k INFO ====> Epoch: 9624, cost 14.46 s 2023-09-05 18:41:47,092 44k INFO ====> Epoch: 9625, cost 14.64 s 2023-09-05 18:42:01,987 44k INFO ====> Epoch: 9626, cost 14.90 s 2023-09-05 18:42:16,511 44k INFO ====> Epoch: 9627, cost 14.52 s 2023-09-05 18:42:30,891 44k INFO ====> Epoch: 9628, cost 14.38 s 2023-09-05 18:42:39,258 44k INFO Train Epoch: 9629 [52%] 2023-09-05 18:42:39,259 44k INFO Losses: [1.9799656867980957, 2.944955587387085, 11.63509750366211, 16.0994873046875, 0.4900258779525757], step: 202200, lr: 3.0011929750555616e-05, reference_loss: 33.149532318115234 2023-09-05 18:42:45,605 44k INFO ====> Epoch: 9629, cost 14.71 s 2023-09-05 18:43:00,186 44k INFO ====> Epoch: 9630, cost 14.58 s 2023-09-05 18:43:14,671 44k INFO ====> Epoch: 9631, cost 14.49 s 2023-09-05 18:43:29,006 44k INFO ====> Epoch: 9632, cost 14.33 s 2023-09-05 18:43:43,546 44k INFO ====> Epoch: 9633, cost 14.54 s 2023-09-05 18:43:58,053 44k INFO ====> Epoch: 9634, cost 14.51 s 2023-09-05 18:44:12,648 44k INFO ====> Epoch: 9635, cost 14.60 s 2023-09-05 18:44:27,225 44k INFO ====> Epoch: 9636, cost 14.58 s 2023-09-05 18:44:41,674 44k INFO ====> Epoch: 9637, cost 14.45 s 2023-09-05 18:44:56,322 44k INFO ====> Epoch: 9638, cost 14.65 s 2023-09-05 18:44:57,727 44k INFO Train Epoch: 9639 [5%] 2023-09-05 18:44:57,727 44k INFO Losses: [2.14009428024292, 2.7593138217926025, 9.43089771270752, 14.930924415588379, 0.6020674109458923], step: 202400, lr: 2.9974435933473003e-05, reference_loss: 29.863298416137695 2023-09-05 18:45:03,183 44k INFO Saving model and optimizer state at iteration 9639 to ./logs\44k\G_202400.pth 2023-09-05 18:45:03,835 44k INFO Saving model and optimizer state at iteration 9639 to ./logs\44k\D_202400.pth 2023-09-05 18:45:04,983 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_200000.pth 2023-09-05 18:45:05,019 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_200000.pth 2023-09-05 18:45:18,014 44k INFO ====> Epoch: 9639, cost 21.69 s 2023-09-05 18:45:32,423 44k INFO ====> Epoch: 9640, cost 14.41 s 2023-09-05 18:45:46,746 44k INFO ====> Epoch: 9641, cost 14.32 s 2023-09-05 18:46:01,403 44k INFO ====> Epoch: 9642, cost 14.66 s 2023-09-05 18:46:15,906 44k INFO ====> Epoch: 9643, cost 14.50 s 2023-09-05 18:46:30,353 44k INFO ====> Epoch: 9644, cost 14.45 s 2023-09-05 18:46:44,907 44k INFO ====> Epoch: 9645, cost 14.55 s 2023-09-05 18:46:59,551 44k INFO ====> Epoch: 9646, cost 14.64 s 2023-09-05 18:47:14,076 44k INFO ====> Epoch: 9647, cost 14.53 s 2023-09-05 18:47:23,239 44k INFO Train Epoch: 9648 [57%] 2023-09-05 18:47:23,240 44k INFO Losses: [2.0084409713745117, 3.1986684799194336, 9.229216575622559, 13.890083312988281, 0.5875938534736633], step: 202600, lr: 2.9940731548751292e-05, reference_loss: 28.91400146484375 2023-09-05 18:47:28,880 44k INFO ====> Epoch: 9648, cost 14.80 s 2023-09-05 18:47:43,412 44k INFO ====> Epoch: 9649, cost 14.53 s 2023-09-05 18:47:58,184 44k INFO ====> Epoch: 9650, cost 14.77 s 2023-09-05 18:48:12,892 44k INFO ====> Epoch: 9651, cost 14.71 s 2023-09-05 18:48:27,337 44k INFO ====> Epoch: 9652, cost 14.44 s 2023-09-05 18:48:41,880 44k INFO ====> Epoch: 9653, cost 14.54 s 2023-09-05 18:48:56,560 44k INFO ====> Epoch: 9654, cost 14.68 s 2023-09-05 18:49:11,139 44k INFO ====> Epoch: 9655, cost 14.58 s 2023-09-05 18:49:25,538 44k INFO ====> Epoch: 9656, cost 14.40 s 2023-09-05 18:49:39,834 44k INFO ====> Epoch: 9657, cost 14.30 s 2023-09-05 18:49:41,973 44k INFO Train Epoch: 9658 [10%] 2023-09-05 18:49:41,973 44k INFO Losses: [2.240405559539795, 2.939453125, 7.759719371795654, 14.121089935302734, 0.550600528717041], step: 202800, lr: 2.990332667937639e-05, reference_loss: 27.611268997192383 2023-09-05 18:49:54,910 44k INFO ====> Epoch: 9658, cost 15.08 s 2023-09-05 18:50:09,678 44k INFO ====> Epoch: 9659, cost 14.77 s 2023-09-05 18:50:24,262 44k INFO ====> Epoch: 9660, cost 14.58 s 2023-09-05 18:50:38,728 44k INFO ====> Epoch: 9661, cost 14.47 s 2023-09-05 18:50:53,273 44k INFO ====> Epoch: 9662, cost 14.55 s 2023-09-05 18:51:07,602 44k INFO ====> Epoch: 9663, cost 14.33 s 2023-09-05 18:51:21,963 44k INFO ====> Epoch: 9664, cost 14.36 s 2023-09-05 18:51:36,321 44k INFO ====> Epoch: 9665, cost 14.36 s 2023-09-05 18:51:50,830 44k INFO ====> Epoch: 9666, cost 14.51 s 2023-09-05 18:52:00,716 44k INFO Train Epoch: 9667 [62%] 2023-09-05 18:52:00,716 44k INFO Losses: [2.1352386474609375, 3.001518726348877, 11.523174285888672, 14.82741641998291, 0.48448511958122253], step: 203000, lr: 2.9869702252578244e-05, reference_loss: 31.971832275390625 2023-09-05 18:52:05,686 44k INFO ====> Epoch: 9667, cost 14.86 s 2023-09-05 18:52:20,298 44k INFO ====> Epoch: 9668, cost 14.61 s 2023-09-05 18:52:34,792 44k INFO ====> Epoch: 9669, cost 14.49 s 2023-09-05 18:52:49,341 44k INFO ====> Epoch: 9670, cost 14.55 s 2023-09-05 18:53:03,841 44k INFO ====> Epoch: 9671, cost 14.50 s 2023-09-05 18:53:18,297 44k INFO ====> Epoch: 9672, cost 14.46 s 2023-09-05 18:53:32,452 44k INFO ====> Epoch: 9673, cost 14.15 s 2023-09-05 18:53:46,952 44k INFO ====> Epoch: 9674, cost 14.50 s 2023-09-05 18:54:01,782 44k INFO ====> Epoch: 9675, cost 14.83 s 2023-09-05 18:54:16,341 44k INFO ====> Epoch: 9676, cost 14.56 s 2023-09-05 18:54:19,138 44k INFO Train Epoch: 9677 [14%] 2023-09-05 18:54:19,138 44k INFO Losses: [1.995926856994629, 3.086200714111328, 12.652215003967285, 14.861993789672852, 0.6446650624275208], step: 203200, lr: 2.9832386119897725e-05, reference_loss: 33.24100112915039 2023-09-05 18:54:24,697 44k INFO Saving model and optimizer state at iteration 9677 to ./logs\44k\G_203200.pth 2023-09-05 18:54:25,244 44k INFO Saving model and optimizer state at iteration 9677 to ./logs\44k\D_203200.pth 2023-09-05 18:54:26,933 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_200800.pth 2023-09-05 18:54:26,965 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_200800.pth 2023-09-05 18:54:38,740 44k INFO ====> Epoch: 9677, cost 22.40 s 2023-09-05 18:54:53,351 44k INFO ====> Epoch: 9678, cost 14.61 s 2023-09-05 18:55:07,874 44k INFO ====> Epoch: 9679, cost 14.52 s 2023-09-05 18:55:22,399 44k INFO ====> Epoch: 9680, cost 14.53 s 2023-09-05 18:55:36,666 44k INFO ====> Epoch: 9681, cost 14.27 s 2023-09-05 18:55:51,298 44k INFO ====> Epoch: 9682, cost 14.63 s 2023-09-05 18:56:05,970 44k INFO ====> Epoch: 9683, cost 14.67 s 2023-09-05 18:56:20,659 44k INFO ====> Epoch: 9684, cost 14.69 s 2023-09-05 18:56:35,267 44k INFO ====> Epoch: 9685, cost 14.61 s 2023-09-05 18:56:45,985 44k INFO Train Epoch: 9686 [67%] 2023-09-05 18:56:45,986 44k INFO Losses: [2.138146162033081, 2.7031242847442627, 9.807767868041992, 15.185214042663574, 0.5142249464988708], step: 203400, lr: 2.979884146133656e-05, reference_loss: 30.34847640991211 2023-09-05 18:56:50,190 44k INFO ====> Epoch: 9686, cost 14.92 s 2023-09-05 18:57:04,690 44k INFO ====> Epoch: 9687, cost 14.50 s 2023-09-05 18:57:19,145 44k INFO ====> Epoch: 9688, cost 14.45 s 2023-09-05 18:57:33,490 44k INFO ====> Epoch: 9689, cost 14.34 s 2023-09-05 18:57:48,054 44k INFO ====> Epoch: 9690, cost 14.56 s 2023-09-05 18:58:02,548 44k INFO ====> Epoch: 9691, cost 14.49 s 2023-09-05 18:58:17,155 44k INFO ====> Epoch: 9692, cost 14.61 s 2023-09-05 18:58:31,494 44k INFO ====> Epoch: 9693, cost 14.34 s 2023-09-05 18:58:46,176 44k INFO ====> Epoch: 9694, cost 14.68 s 2023-09-05 18:59:00,744 44k INFO ====> Epoch: 9695, cost 14.57 s 2023-09-05 18:59:04,352 44k INFO Train Epoch: 9696 [19%] 2023-09-05 18:59:04,352 44k INFO Losses: [2.427919626235962, 2.4306116104125977, 8.436004638671875, 15.508564949035645, 0.4866863787174225], step: 203600, lr: 2.9761613854837707e-05, reference_loss: 29.28978729248047 2023-09-05 18:59:15,718 44k INFO ====> Epoch: 9696, cost 14.97 s 2023-09-05 18:59:30,110 44k INFO ====> Epoch: 9697, cost 14.39 s 2023-09-05 18:59:44,573 44k INFO ====> Epoch: 9698, cost 14.46 s 2023-09-05 18:59:59,352 44k INFO ====> Epoch: 9699, cost 14.78 s 2023-09-05 19:00:14,003 44k INFO ====> Epoch: 9700, cost 14.65 s 2023-09-05 19:00:28,553 44k INFO ====> Epoch: 9701, cost 14.55 s 2023-09-05 19:00:43,074 44k INFO ====> Epoch: 9702, cost 14.52 s 2023-09-05 19:00:57,656 44k INFO ====> Epoch: 9703, cost 14.58 s 2023-09-05 19:01:12,030 44k INFO ====> Epoch: 9704, cost 14.37 s 2023-09-05 19:01:23,248 44k INFO Train Epoch: 9705 [71%] 2023-09-05 19:01:23,248 44k INFO Losses: [2.136117935180664, 2.922603130340576, 9.8859224319458, 13.66321849822998, 0.4414883553981781], step: 203800, lr: 2.972814877527695e-05, reference_loss: 29.049348831176758 2023-09-05 19:01:26,818 44k INFO ====> Epoch: 9705, cost 14.79 s 2023-09-05 19:01:41,240 44k INFO ====> Epoch: 9706, cost 14.42 s 2023-09-05 19:01:55,837 44k INFO ====> Epoch: 9707, cost 14.60 s 2023-09-05 19:02:10,505 44k INFO ====> Epoch: 9708, cost 14.67 s 2023-09-05 19:02:24,852 44k INFO ====> Epoch: 9709, cost 14.35 s 2023-09-05 19:02:39,316 44k INFO ====> Epoch: 9710, cost 14.46 s 2023-09-05 19:02:54,007 44k INFO ====> Epoch: 9711, cost 14.69 s 2023-09-05 19:03:08,400 44k INFO ====> Epoch: 9712, cost 14.39 s 2023-09-05 19:03:22,772 44k INFO ====> Epoch: 9713, cost 14.37 s 2023-09-05 19:03:37,175 44k INFO ====> Epoch: 9714, cost 14.40 s 2023-09-05 19:03:41,494 44k INFO Train Epoch: 9715 [24%] 2023-09-05 19:03:41,495 44k INFO Losses: [2.1978864669799805, 2.74674391746521, 8.964237213134766, 14.78775691986084, 0.6900460720062256], step: 204000, lr: 2.9691009484946436e-05, reference_loss: 29.38667106628418 2023-09-05 19:03:47,019 44k INFO Saving model and optimizer state at iteration 9715 to ./logs\44k\G_204000.pth 2023-09-05 19:03:47,576 44k INFO Saving model and optimizer state at iteration 9715 to ./logs\44k\D_204000.pth 2023-09-05 19:03:49,247 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_201600.pth 2023-09-05 19:03:49,285 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_201600.pth 2023-09-05 19:03:59,554 44k INFO ====> Epoch: 9715, cost 22.38 s 2023-09-05 19:04:14,186 44k INFO ====> Epoch: 9716, cost 14.63 s 2023-09-05 19:04:28,823 44k INFO ====> Epoch: 9717, cost 14.64 s 2023-09-05 19:04:43,338 44k INFO ====> Epoch: 9718, cost 14.52 s 2023-09-05 19:04:57,949 44k INFO ====> Epoch: 9719, cost 14.61 s 2023-09-05 19:05:12,710 44k INFO ====> Epoch: 9720, cost 14.76 s 2023-09-05 19:05:27,164 44k INFO ====> Epoch: 9721, cost 14.45 s 2023-09-05 19:05:41,591 44k INFO ====> Epoch: 9722, cost 14.43 s 2023-09-05 19:05:56,343 44k INFO ====> Epoch: 9723, cost 14.75 s 2023-09-05 19:06:08,500 44k INFO Train Epoch: 9724 [76%] 2023-09-05 19:06:08,501 44k INFO Losses: [2.109837293624878, 2.4815492630004883, 10.279146194458008, 15.977972984313965, 0.6912823915481567], step: 204200, lr: 2.965762379559843e-05, reference_loss: 31.5397891998291 2023-09-05 19:06:11,426 44k INFO ====> Epoch: 9724, cost 15.08 s 2023-09-05 19:06:25,773 44k INFO ====> Epoch: 9725, cost 14.35 s 2023-09-05 19:06:40,155 44k INFO ====> Epoch: 9726, cost 14.38 s 2023-09-05 19:06:54,732 44k INFO ====> Epoch: 9727, cost 14.58 s 2023-09-05 19:07:09,468 44k INFO ====> Epoch: 9728, cost 14.74 s 2023-09-05 19:07:23,963 44k INFO ====> Epoch: 9729, cost 14.50 s 2023-09-05 19:07:38,395 44k INFO ====> Epoch: 9730, cost 14.43 s 2023-09-05 19:07:52,995 44k INFO ====> Epoch: 9731, cost 14.60 s 2023-09-05 19:08:07,810 44k INFO ====> Epoch: 9732, cost 14.81 s 2023-09-05 19:08:22,343 44k INFO ====> Epoch: 9733, cost 14.53 s 2023-09-05 19:08:27,296 44k INFO Train Epoch: 9734 [29%] 2023-09-05 19:08:27,297 44k INFO Losses: [2.0848515033721924, 3.0683107376098633, 9.580265045166016, 15.107401847839355, 0.5049421787261963], step: 204400, lr: 2.9620572611921164e-05, reference_loss: 30.34576988220215 2023-09-05 19:08:37,185 44k INFO ====> Epoch: 9734, cost 14.84 s 2023-09-05 19:08:51,691 44k INFO ====> Epoch: 9735, cost 14.51 s 2023-09-05 19:09:06,392 44k INFO ====> Epoch: 9736, cost 14.70 s 2023-09-05 19:09:20,749 44k INFO ====> Epoch: 9737, cost 14.36 s 2023-09-05 19:09:35,036 44k INFO ====> Epoch: 9738, cost 14.29 s 2023-09-05 19:09:49,570 44k INFO ====> Epoch: 9739, cost 14.53 s 2023-09-05 19:10:04,155 44k INFO ====> Epoch: 9740, cost 14.58 s 2023-09-05 19:10:18,653 44k INFO ====> Epoch: 9741, cost 14.50 s 2023-09-05 19:10:33,113 44k INFO ====> Epoch: 9742, cost 14.46 s 2023-09-05 19:10:45,779 44k INFO Train Epoch: 9743 [81%] 2023-09-05 19:10:45,780 44k INFO Losses: [2.1181516647338867, 2.609750747680664, 8.867562294006348, 14.284160614013672, 0.6762400269508362], step: 204600, lr: 2.9587266124446125e-05, reference_loss: 28.555866241455078 2023-09-05 19:10:47,986 44k INFO ====> Epoch: 9743, cost 14.87 s 2023-09-05 19:11:02,699 44k INFO ====> Epoch: 9744, cost 14.71 s 2023-09-05 19:11:17,193 44k INFO ====> Epoch: 9745, cost 14.49 s 2023-09-05 19:11:31,528 44k INFO ====> Epoch: 9746, cost 14.33 s 2023-09-05 19:11:45,863 44k INFO ====> Epoch: 9747, cost 14.34 s 2023-09-05 19:12:00,658 44k INFO ====> Epoch: 9748, cost 14.79 s 2023-09-05 19:12:15,268 44k INFO ====> Epoch: 9749, cost 14.61 s 2023-09-05 19:12:29,969 44k INFO ====> Epoch: 9750, cost 14.70 s 2023-09-05 19:12:44,488 44k INFO ====> Epoch: 9751, cost 14.52 s 2023-09-05 19:12:59,060 44k INFO ====> Epoch: 9752, cost 14.57 s 2023-09-05 19:13:04,686 44k INFO Train Epoch: 9753 [33%] 2023-09-05 19:13:04,687 44k INFO Losses: [1.9987256526947021, 2.9632461071014404, 10.167900085449219, 14.164487838745117, 0.6229273676872253], step: 204800, lr: 2.955030283840405e-05, reference_loss: 29.917285919189453 2023-09-05 19:13:10,172 44k INFO Saving model and optimizer state at iteration 9753 to ./logs\44k\G_204800.pth 2023-09-05 19:13:10,910 44k INFO Saving model and optimizer state at iteration 9753 to ./logs\44k\D_204800.pth 2023-09-05 19:13:12,034 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_202400.pth 2023-09-05 19:13:12,070 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_202400.pth 2023-09-05 19:13:20,838 44k INFO ====> Epoch: 9753, cost 21.78 s 2023-09-05 19:13:35,257 44k INFO ====> Epoch: 9754, cost 14.42 s 2023-09-05 19:13:49,740 44k INFO ====> Epoch: 9755, cost 14.48 s 2023-09-05 19:14:04,502 44k INFO ====> Epoch: 9756, cost 14.76 s 2023-09-05 19:14:18,948 44k INFO ====> Epoch: 9757, cost 14.45 s 2023-09-05 19:14:33,473 44k INFO ====> Epoch: 9758, cost 14.52 s 2023-09-05 19:14:47,883 44k INFO ====> Epoch: 9759, cost 14.41 s 2023-09-05 19:15:02,504 44k INFO ====> Epoch: 9760, cost 14.62 s 2023-09-05 19:15:17,070 44k INFO ====> Epoch: 9761, cost 14.57 s 2023-09-05 19:15:30,310 44k INFO Train Epoch: 9762 [86%] 2023-09-05 19:15:30,310 44k INFO Losses: [1.916747808456421, 2.947983741760254, 12.92022705078125, 15.302650451660156, 0.7184885144233704], step: 205000, lr: 2.9517075364908997e-05, reference_loss: 33.806095123291016 2023-09-05 19:15:31,712 44k INFO ====> Epoch: 9762, cost 14.64 s 2023-09-05 19:15:46,081 44k INFO ====> Epoch: 9763, cost 14.37 s 2023-09-05 19:16:00,922 44k INFO ====> Epoch: 9764, cost 14.84 s 2023-09-05 19:16:15,439 44k INFO ====> Epoch: 9765, cost 14.52 s 2023-09-05 19:16:29,791 44k INFO ====> Epoch: 9766, cost 14.35 s 2023-09-05 19:16:44,362 44k INFO ====> Epoch: 9767, cost 14.57 s 2023-09-05 19:16:58,954 44k INFO ====> Epoch: 9768, cost 14.59 s 2023-09-05 19:17:13,667 44k INFO ====> Epoch: 9769, cost 14.71 s 2023-09-05 19:17:28,111 44k INFO ====> Epoch: 9770, cost 14.44 s 2023-09-05 19:17:42,514 44k INFO ====> Epoch: 9771, cost 14.40 s 2023-09-05 19:17:48,954 44k INFO Train Epoch: 9772 [38%] 2023-09-05 19:17:48,955 44k INFO Losses: [1.8007774353027344, 3.06911301612854, 11.395649909973145, 15.405577659606934, 0.530663013458252], step: 205200, lr: 2.9480199767979918e-05, reference_loss: 32.2017822265625 2023-09-05 19:17:57,487 44k INFO ====> Epoch: 9772, cost 14.97 s 2023-09-05 19:18:12,254 44k INFO ====> Epoch: 9773, cost 14.77 s 2023-09-05 19:18:26,668 44k INFO ====> Epoch: 9774, cost 14.41 s 2023-09-05 19:18:41,361 44k INFO ====> Epoch: 9775, cost 14.69 s 2023-09-05 19:18:56,025 44k INFO ====> Epoch: 9776, cost 14.66 s 2023-09-05 19:19:10,566 44k INFO ====> Epoch: 9777, cost 14.54 s 2023-09-05 19:19:25,023 44k INFO ====> Epoch: 9778, cost 14.46 s 2023-09-05 19:19:39,218 44k INFO ====> Epoch: 9779, cost 14.19 s 2023-09-05 19:19:53,923 44k INFO ====> Epoch: 9780, cost 14.71 s 2023-09-05 19:20:07,932 44k INFO Train Epoch: 9781 [90%] 2023-09-05 19:20:07,933 44k INFO Losses: [2.149263381958008, 2.6682446002960205, 11.840087890625, 16.08938980102539, 0.4015331566333771], step: 205400, lr: 2.9447051121017607e-05, reference_loss: 33.148521423339844 2023-09-05 19:20:08,642 44k INFO ====> Epoch: 9781, cost 14.72 s 2023-09-05 19:20:23,358 44k INFO ====> Epoch: 9782, cost 14.72 s 2023-09-05 19:20:37,879 44k INFO ====> Epoch: 9783, cost 14.52 s 2023-09-05 19:20:52,316 44k INFO ====> Epoch: 9784, cost 14.44 s 2023-09-05 19:21:06,843 44k INFO ====> Epoch: 9785, cost 14.53 s 2023-09-05 19:21:21,347 44k INFO ====> Epoch: 9786, cost 14.50 s 2023-09-05 19:21:35,733 44k INFO ====> Epoch: 9787, cost 14.39 s 2023-09-05 19:21:50,306 44k INFO ====> Epoch: 9788, cost 14.57 s 2023-09-05 19:22:04,895 44k INFO ====> Epoch: 9789, cost 14.59 s 2023-09-05 19:22:19,388 44k INFO ====> Epoch: 9790, cost 14.49 s 2023-09-05 19:22:26,366 44k INFO Train Epoch: 9791 [43%] 2023-09-05 19:22:26,366 44k INFO Losses: [2.273561477661133, 2.423518657684326, 9.601249694824219, 14.480937957763672, 0.6639825105667114], step: 205600, lr: 2.9410263005174e-05, reference_loss: 29.44325065612793 2023-09-05 19:22:31,947 44k INFO Saving model and optimizer state at iteration 9791 to ./logs\44k\G_205600.pth 2023-09-05 19:22:32,687 44k INFO Saving model and optimizer state at iteration 9791 to ./logs\44k\D_205600.pth 2023-09-05 19:22:33,848 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_203200.pth 2023-09-05 19:22:33,886 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_203200.pth 2023-09-05 19:22:41,307 44k INFO ====> Epoch: 9791, cost 21.92 s 2023-09-05 19:22:55,874 44k INFO ====> Epoch: 9792, cost 14.57 s 2023-09-05 19:23:10,440 44k INFO ====> Epoch: 9793, cost 14.57 s 2023-09-05 19:23:24,682 44k INFO ====> Epoch: 9794, cost 14.24 s 2023-09-05 19:23:39,015 44k INFO ====> Epoch: 9795, cost 14.33 s 2023-09-05 19:23:53,818 44k INFO ====> Epoch: 9796, cost 14.80 s 2023-09-05 19:24:08,673 44k INFO ====> Epoch: 9797, cost 14.86 s 2023-09-05 19:24:23,221 44k INFO ====> Epoch: 9798, cost 14.55 s 2023-09-05 19:24:37,768 44k INFO ====> Epoch: 9799, cost 14.55 s 2023-09-05 19:24:52,330 44k INFO Train Epoch: 9800 [95%] 2023-09-05 19:24:52,330 44k INFO Losses: [2.0532968044281006, 2.9392828941345215, 9.130504608154297, 13.41767406463623, 0.5500685572624207], step: 205800, lr: 2.937719299774189e-05, reference_loss: 28.0908260345459 2023-09-05 19:24:52,595 44k INFO ====> Epoch: 9800, cost 14.83 s 2023-09-05 19:25:07,033 44k INFO ====> Epoch: 9801, cost 14.44 s 2023-09-05 19:25:21,512 44k INFO ====> Epoch: 9802, cost 14.48 s 2023-09-05 19:25:35,764 44k INFO ====> Epoch: 9803, cost 14.25 s 2023-09-05 19:25:50,121 44k INFO ====> Epoch: 9804, cost 14.36 s 2023-09-05 19:26:04,804 44k INFO ====> Epoch: 9805, cost 14.68 s 2023-09-05 19:26:19,459 44k INFO ====> Epoch: 9806, cost 14.65 s 2023-09-05 19:26:34,106 44k INFO ====> Epoch: 9807, cost 14.65 s 2023-09-05 19:26:48,542 44k INFO ====> Epoch: 9808, cost 14.44 s 2023-09-05 19:27:03,376 44k INFO ====> Epoch: 9809, cost 14.83 s 2023-09-05 19:27:11,179 44k INFO Train Epoch: 9810 [48%] 2023-09-05 19:27:11,180 44k INFO Losses: [2.1345949172973633, 2.752609968185425, 11.199088096618652, 16.070507049560547, 0.6086174845695496], step: 206000, lr: 2.9340492155449756e-05, reference_loss: 32.76541519165039 2023-09-05 19:27:18,148 44k INFO ====> Epoch: 9810, cost 14.77 s 2023-09-05 19:27:32,493 44k INFO ====> Epoch: 9811, cost 14.35 s 2023-09-05 19:27:46,951 44k INFO ====> Epoch: 9812, cost 14.46 s 2023-09-05 19:28:01,625 44k INFO ====> Epoch: 9813, cost 14.67 s 2023-09-05 19:28:16,236 44k INFO ====> Epoch: 9814, cost 14.61 s 2023-09-05 19:28:30,881 44k INFO ====> Epoch: 9815, cost 14.64 s 2023-09-05 19:28:45,328 44k INFO ====> Epoch: 9816, cost 14.45 s 2023-09-05 19:28:59,952 44k INFO ====> Epoch: 9817, cost 14.62 s 2023-09-05 19:29:14,545 44k INFO ====> Epoch: 9818, cost 14.59 s 2023-09-05 19:29:28,811 44k INFO ====> Epoch: 9819, cost 14.27 s 2023-09-05 19:29:29,493 44k INFO Train Epoch: 9820 [0%] 2023-09-05 19:29:29,493 44k INFO Losses: [2.2262816429138184, 2.788818836212158, 9.802027702331543, 14.097637176513672, 0.6517945528030396], step: 206200, lr: 2.9303837163413808e-05, reference_loss: 29.566558837890625 2023-09-05 19:29:43,476 44k INFO ====> Epoch: 9820, cost 14.66 s 2023-09-05 19:29:58,174 44k INFO ====> Epoch: 9821, cost 14.70 s 2023-09-05 19:30:12,779 44k INFO ====> Epoch: 9822, cost 14.61 s 2023-09-05 19:30:27,291 44k INFO ====> Epoch: 9823, cost 14.51 s 2023-09-05 19:30:41,764 44k INFO ====> Epoch: 9824, cost 14.47 s 2023-09-05 19:30:56,515 44k INFO ====> Epoch: 9825, cost 14.75 s 2023-09-05 19:31:11,191 44k INFO ====> Epoch: 9826, cost 14.68 s 2023-09-05 19:31:25,474 44k INFO ====> Epoch: 9827, cost 14.28 s 2023-09-05 19:31:39,868 44k INFO ====> Epoch: 9828, cost 14.39 s 2023-09-05 19:31:48,320 44k INFO Train Epoch: 9829 [52%] 2023-09-05 19:31:48,320 44k INFO Losses: [2.0767621994018555, 2.9965808391571045, 12.001814842224121, 15.367446899414062, 0.6376570463180542], step: 206400, lr: 2.9270886825206595e-05, reference_loss: 33.08026123046875 2023-09-05 19:31:53,799 44k INFO Saving model and optimizer state at iteration 9829 to ./logs\44k\G_206400.pth 2023-09-05 19:31:54,487 44k INFO Saving model and optimizer state at iteration 9829 to ./logs\44k\D_206400.pth 2023-09-05 19:31:55,641 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_204000.pth 2023-09-05 19:31:55,680 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_204000.pth 2023-09-05 19:32:01,783 44k INFO ====> Epoch: 9829, cost 21.91 s 2023-09-05 19:32:16,440 44k INFO ====> Epoch: 9830, cost 14.66 s 2023-09-05 19:32:30,978 44k INFO ====> Epoch: 9831, cost 14.54 s 2023-09-05 19:32:45,607 44k INFO ====> Epoch: 9832, cost 14.63 s 2023-09-05 19:33:00,272 44k INFO ====> Epoch: 9833, cost 14.66 s 2023-09-05 19:33:14,780 44k INFO ====> Epoch: 9834, cost 14.51 s 2023-09-05 19:33:29,186 44k INFO ====> Epoch: 9835, cost 14.41 s 2023-09-05 19:33:43,635 44k INFO ====> Epoch: 9836, cost 14.45 s 2023-09-05 19:33:58,134 44k INFO ====> Epoch: 9837, cost 14.50 s 2023-09-05 19:34:12,680 44k INFO ====> Epoch: 9838, cost 14.55 s 2023-09-05 19:34:14,105 44k INFO Train Epoch: 9839 [5%] 2023-09-05 19:34:14,105 44k INFO Losses: [2.12734317779541, 2.955880880355835, 9.770293235778809, 13.356148719787598, 0.6137747168540955], step: 206600, lr: 2.9234318790908512e-05, reference_loss: 28.823442459106445 2023-09-05 19:34:27,396 44k INFO ====> Epoch: 9839, cost 14.72 s 2023-09-05 19:34:41,795 44k INFO ====> Epoch: 9840, cost 14.40 s 2023-09-05 19:34:56,371 44k INFO ====> Epoch: 9841, cost 14.58 s 2023-09-05 19:35:11,041 44k INFO ====> Epoch: 9842, cost 14.67 s 2023-09-05 19:35:25,445 44k INFO ====> Epoch: 9843, cost 14.40 s 2023-09-05 19:35:39,843 44k INFO ====> Epoch: 9844, cost 14.40 s 2023-09-05 19:35:54,684 44k INFO ====> Epoch: 9845, cost 14.84 s 2023-09-05 19:36:09,323 44k INFO ====> Epoch: 9846, cost 14.64 s 2023-09-05 19:36:23,843 44k INFO ====> Epoch: 9847, cost 14.52 s 2023-09-05 19:36:32,967 44k INFO Train Epoch: 9848 [57%] 2023-09-05 19:36:32,967 44k INFO Losses: [2.3426170349121094, 2.497687816619873, 9.563429832458496, 16.133974075317383, 0.4837978184223175], step: 206800, lr: 2.920144662177769e-05, reference_loss: 31.02150535583496 2023-09-05 19:36:38,542 44k INFO ====> Epoch: 9848, cost 14.70 s 2023-09-05 19:36:53,258 44k INFO ====> Epoch: 9849, cost 14.72 s 2023-09-05 19:37:07,842 44k INFO ====> Epoch: 9850, cost 14.58 s 2023-09-05 19:37:22,346 44k INFO ====> Epoch: 9851, cost 14.50 s 2023-09-05 19:37:36,664 44k INFO ====> Epoch: 9852, cost 14.32 s 2023-09-05 19:37:51,234 44k INFO ====> Epoch: 9853, cost 14.57 s 2023-09-05 19:38:05,898 44k INFO ====> Epoch: 9854, cost 14.66 s 2023-09-05 19:38:20,291 44k INFO ====> Epoch: 9855, cost 14.39 s 2023-09-05 19:38:35,012 44k INFO ====> Epoch: 9856, cost 14.72 s 2023-09-05 19:38:49,604 44k INFO ====> Epoch: 9857, cost 14.59 s 2023-09-05 19:38:51,687 44k INFO Train Epoch: 9858 [10%] 2023-09-05 19:38:51,688 44k INFO Losses: [2.129706382751465, 2.817269802093506, 8.982077598571777, 13.552695274353027, 0.5874366760253906], step: 207000, lr: 2.9164965338925025e-05, reference_loss: 28.06918716430664 2023-09-05 19:39:04,568 44k INFO ====> Epoch: 9858, cost 14.96 s 2023-09-05 19:39:19,055 44k INFO ====> Epoch: 9859, cost 14.49 s 2023-09-05 19:39:33,291 44k INFO ====> Epoch: 9860, cost 14.24 s 2023-09-05 19:39:47,938 44k INFO ====> Epoch: 9861, cost 14.65 s 2023-09-05 19:40:02,474 44k INFO ====> Epoch: 9862, cost 14.54 s 2023-09-05 19:40:17,066 44k INFO ====> Epoch: 9863, cost 14.59 s 2023-09-05 19:40:31,494 44k INFO ====> Epoch: 9864, cost 14.43 s 2023-09-05 19:40:46,221 44k INFO ====> Epoch: 9865, cost 14.73 s 2023-09-05 19:41:00,810 44k INFO ====> Epoch: 9866, cost 14.59 s 2023-09-05 19:41:10,777 44k INFO Train Epoch: 9867 [62%] 2023-09-05 19:41:10,778 44k INFO Losses: [2.3381717205047607, 2.395336866378784, 8.09886360168457, 13.975037574768066, 0.5664407014846802], step: 207200, lr: 2.9132171153427743e-05, reference_loss: 27.373851776123047 2023-09-05 19:41:16,299 44k INFO Saving model and optimizer state at iteration 9867 to ./logs\44k\G_207200.pth 2023-09-05 19:41:16,970 44k INFO Saving model and optimizer state at iteration 9867 to ./logs\44k\D_207200.pth 2023-09-05 19:41:18,548 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_204800.pth 2023-09-05 19:41:18,584 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_204800.pth 2023-09-05 19:41:23,268 44k INFO ====> Epoch: 9867, cost 22.46 s 2023-09-05 19:41:37,682 44k INFO ====> Epoch: 9868, cost 14.41 s 2023-09-05 19:41:52,245 44k INFO ====> Epoch: 9869, cost 14.56 s 2023-09-05 19:42:07,091 44k INFO ====> Epoch: 9870, cost 14.85 s 2023-09-05 19:42:21,534 44k INFO ====> Epoch: 9871, cost 14.44 s 2023-09-05 19:42:36,017 44k INFO ====> Epoch: 9872, cost 14.48 s 2023-09-05 19:42:50,764 44k INFO ====> Epoch: 9873, cost 14.75 s 2023-09-05 19:43:05,342 44k INFO ====> Epoch: 9874, cost 14.58 s 2023-09-05 19:43:19,766 44k INFO ====> Epoch: 9875, cost 14.42 s 2023-09-05 19:43:34,113 44k INFO ====> Epoch: 9876, cost 14.35 s 2023-09-05 19:43:36,940 44k INFO Train Epoch: 9877 [14%] 2023-09-05 19:43:36,940 44k INFO Losses: [2.089604377746582, 2.9166836738586426, 9.630483627319336, 16.695707321166992, 0.6970880627632141], step: 207400, lr: 2.909577641621743e-05, reference_loss: 32.02956771850586 2023-09-05 19:43:48,884 44k INFO ====> Epoch: 9877, cost 14.77 s 2023-09-05 19:44:03,650 44k INFO ====> Epoch: 9878, cost 14.77 s 2023-09-05 19:44:18,194 44k INFO ====> Epoch: 9879, cost 14.54 s 2023-09-05 19:44:32,582 44k INFO ====> Epoch: 9880, cost 14.39 s 2023-09-05 19:44:47,122 44k INFO ====> Epoch: 9881, cost 14.54 s 2023-09-05 19:45:01,637 44k INFO ====> Epoch: 9882, cost 14.52 s 2023-09-05 19:45:16,088 44k INFO ====> Epoch: 9883, cost 14.45 s 2023-09-05 19:45:30,405 44k INFO ====> Epoch: 9884, cost 14.32 s 2023-09-05 19:45:44,843 44k INFO ====> Epoch: 9885, cost 14.44 s 2023-09-05 19:45:55,475 44k INFO Train Epoch: 9886 [67%] 2023-09-05 19:45:55,475 44k INFO Losses: [2.1179039478302, 2.8961243629455566, 9.671798706054688, 15.891185760498047, 0.5437218546867371], step: 207600, lr: 2.9063060029350776e-05, reference_loss: 31.12073516845703 2023-09-05 19:45:59,733 44k INFO ====> Epoch: 9886, cost 14.89 s 2023-09-05 19:46:14,321 44k INFO ====> Epoch: 9887, cost 14.59 s 2023-09-05 19:46:28,866 44k INFO ====> Epoch: 9888, cost 14.55 s 2023-09-05 19:46:43,355 44k INFO ====> Epoch: 9889, cost 14.49 s 2023-09-05 19:46:57,980 44k INFO ====> Epoch: 9890, cost 14.63 s 2023-09-05 19:47:12,511 44k INFO ====> Epoch: 9891, cost 14.53 s 2023-09-05 19:47:26,909 44k INFO ====> Epoch: 9892, cost 14.40 s 2023-09-05 19:47:41,509 44k INFO ====> Epoch: 9893, cost 14.60 s 2023-09-05 19:47:56,214 44k INFO ====> Epoch: 9894, cost 14.70 s 2023-09-05 19:48:10,981 44k INFO ====> Epoch: 9895, cost 14.77 s 2023-09-05 19:48:14,546 44k INFO Train Epoch: 9896 [19%] 2023-09-05 19:48:14,546 44k INFO Losses: [2.073711633682251, 2.84761381149292, 9.686441421508789, 12.97984504699707, 0.48870599269866943], step: 207800, lr: 2.9026751632468e-05, reference_loss: 28.076318740844727 2023-09-05 19:48:25,772 44k INFO ====> Epoch: 9896, cost 14.79 s 2023-09-05 19:48:40,301 44k INFO ====> Epoch: 9897, cost 14.53 s 2023-09-05 19:48:54,844 44k INFO ====> Epoch: 9898, cost 14.54 s 2023-09-05 19:49:09,450 44k INFO ====> Epoch: 9899, cost 14.61 s 2023-09-05 19:49:23,891 44k INFO ====> Epoch: 9900, cost 14.44 s 2023-09-05 19:49:38,345 44k INFO ====> Epoch: 9901, cost 14.45 s 2023-09-05 19:49:53,070 44k INFO ====> Epoch: 9902, cost 14.72 s 2023-09-05 19:50:07,975 44k INFO ====> Epoch: 9903, cost 14.91 s 2023-09-05 19:50:22,395 44k INFO ====> Epoch: 9904, cost 14.42 s 2023-09-05 19:50:33,599 44k INFO Train Epoch: 9905 [71%] 2023-09-05 19:50:33,599 44k INFO Losses: [2.054631471633911, 2.867579936981201, 11.307369232177734, 15.034500122070312, 0.5996984243392944], step: 208000, lr: 2.899411285966795e-05, reference_loss: 31.863779067993164 2023-09-05 19:50:39,105 44k INFO Saving model and optimizer state at iteration 9905 to ./logs\44k\G_208000.pth 2023-09-05 19:50:39,777 44k INFO Saving model and optimizer state at iteration 9905 to ./logs\44k\D_208000.pth 2023-09-05 19:50:40,939 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_205600.pth 2023-09-05 19:50:40,978 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_205600.pth 2023-09-05 19:50:44,224 44k INFO ====> Epoch: 9905, cost 21.83 s 2023-09-05 19:50:58,676 44k INFO ====> Epoch: 9906, cost 14.45 s 2023-09-05 19:51:13,127 44k INFO ====> Epoch: 9907, cost 14.45 s 2023-09-05 19:51:27,632 44k INFO ====> Epoch: 9908, cost 14.51 s 2023-09-05 19:51:41,943 44k INFO ====> Epoch: 9909, cost 14.31 s 2023-09-05 19:51:56,467 44k INFO ====> Epoch: 9910, cost 14.52 s 2023-09-05 19:52:11,191 44k INFO ====> Epoch: 9911, cost 14.72 s 2023-09-05 19:52:25,574 44k INFO ====> Epoch: 9912, cost 14.38 s 2023-09-05 19:52:40,054 44k INFO ====> Epoch: 9913, cost 14.48 s 2023-09-05 19:52:54,645 44k INFO ====> Epoch: 9914, cost 14.59 s 2023-09-05 19:52:58,949 44k INFO Train Epoch: 9915 [24%] 2023-09-05 19:52:58,950 44k INFO Losses: [2.3491244316101074, 2.744034767150879, 9.37980842590332, 13.84138298034668, 0.5981084108352661], step: 208200, lr: 2.895789059828495e-05, reference_loss: 28.912458419799805 2023-09-05 19:53:09,506 44k INFO ====> Epoch: 9915, cost 14.86 s 2023-09-05 19:53:23,884 44k INFO ====> Epoch: 9916, cost 14.38 s 2023-09-05 19:53:38,305 44k INFO ====> Epoch: 9917, cost 14.42 s 2023-09-05 19:53:52,966 44k INFO ====> Epoch: 9918, cost 14.66 s 2023-09-05 19:54:07,664 44k INFO ====> Epoch: 9919, cost 14.70 s 2023-09-05 19:54:22,112 44k INFO ====> Epoch: 9920, cost 14.45 s 2023-09-05 19:54:36,557 44k INFO ====> Epoch: 9921, cost 14.45 s 2023-09-05 19:54:50,980 44k INFO ====> Epoch: 9922, cost 14.42 s 2023-09-05 19:55:05,658 44k INFO ====> Epoch: 9923, cost 14.68 s 2023-09-05 19:55:17,409 44k INFO Train Epoch: 9924 [76%] 2023-09-05 19:55:17,409 44k INFO Losses: [2.0494773387908936, 2.8957905769348145, 11.870209693908691, 16.233680725097656, 0.5473079681396484], step: 208400, lr: 2.892532925542532e-05, reference_loss: 33.596466064453125 2023-09-05 19:55:20,174 44k INFO ====> Epoch: 9924, cost 14.52 s 2023-09-05 19:55:34,606 44k INFO ====> Epoch: 9925, cost 14.43 s 2023-09-05 19:55:48,939 44k INFO ====> Epoch: 9926, cost 14.33 s 2023-09-05 19:56:03,751 44k INFO ====> Epoch: 9927, cost 14.81 s 2023-09-05 19:56:18,203 44k INFO ====> Epoch: 9928, cost 14.45 s 2023-09-05 19:56:32,961 44k INFO ====> Epoch: 9929, cost 14.76 s 2023-09-05 19:56:47,524 44k INFO ====> Epoch: 9930, cost 14.56 s 2023-09-05 19:57:02,070 44k INFO ====> Epoch: 9931, cost 14.55 s 2023-09-05 19:57:16,440 44k INFO ====> Epoch: 9932, cost 14.37 s 2023-09-05 19:57:30,823 44k INFO ====> Epoch: 9933, cost 14.38 s 2023-09-05 19:57:35,732 44k INFO Train Epoch: 9934 [29%] 2023-09-05 19:57:35,732 44k INFO Losses: [2.105010986328125, 2.639437198638916, 10.15170669555664, 13.273465156555176, 0.46684616804122925], step: 208600, lr: 2.8889192925200267e-05, reference_loss: 28.63646697998047 2023-09-05 19:57:45,556 44k INFO ====> Epoch: 9934, cost 14.73 s 2023-09-05 19:58:00,225 44k INFO ====> Epoch: 9935, cost 14.67 s 2023-09-05 19:58:14,917 44k INFO ====> Epoch: 9936, cost 14.69 s 2023-09-05 19:58:29,358 44k INFO ====> Epoch: 9937, cost 14.44 s 2023-09-05 19:58:43,805 44k INFO ====> Epoch: 9938, cost 14.45 s 2023-09-05 19:58:58,633 44k INFO ====> Epoch: 9939, cost 14.83 s 2023-09-05 19:59:13,089 44k INFO ====> Epoch: 9940, cost 14.46 s 2023-09-05 19:59:27,437 44k INFO ====> Epoch: 9941, cost 14.35 s 2023-09-05 19:59:41,670 44k INFO ====> Epoch: 9942, cost 14.23 s 2023-09-05 19:59:54,800 44k INFO Train Epoch: 9943 [81%] 2023-09-05 19:59:54,800 44k INFO Losses: [2.063366651535034, 2.9679696559906006, 10.176410675048828, 13.840657234191895, 0.5521490573883057], step: 208800, lr: 2.885670882859168e-05, reference_loss: 29.600553512573242 2023-09-05 20:00:00,416 44k INFO Saving model and optimizer state at iteration 9943 to ./logs\44k\G_208800.pth 2023-09-05 20:00:00,980 44k INFO Saving model and optimizer state at iteration 9943 to ./logs\44k\D_208800.pth 2023-09-05 20:00:02,584 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_206400.pth 2023-09-05 20:00:02,621 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_206400.pth 2023-09-05 20:00:04,495 44k INFO ====> Epoch: 9943, cost 22.82 s 2023-09-05 20:00:19,208 44k INFO ====> Epoch: 9944, cost 14.71 s 2023-09-05 20:00:33,677 44k INFO ====> Epoch: 9945, cost 14.47 s 2023-09-05 20:00:48,183 44k INFO ====> Epoch: 9946, cost 14.51 s 2023-09-05 20:01:02,984 44k INFO ====> Epoch: 9947, cost 14.80 s 2023-09-05 20:01:17,629 44k INFO ====> Epoch: 9948, cost 14.64 s 2023-09-05 20:01:32,044 44k INFO ====> Epoch: 9949, cost 14.42 s 2023-09-05 20:01:46,555 44k INFO ====> Epoch: 9950, cost 14.51 s 2023-09-05 20:02:01,288 44k INFO ====> Epoch: 9951, cost 14.73 s 2023-09-05 20:02:15,915 44k INFO ====> Epoch: 9952, cost 14.63 s 2023-09-05 20:02:21,527 44k INFO Train Epoch: 9953 [33%] 2023-09-05 20:02:21,528 44k INFO Losses: [2.0545754432678223, 3.1419143676757812, 12.114981651306152, 14.965787887573242, 0.6020843386650085], step: 209000, lr: 2.8820658225667516e-05, reference_loss: 32.87934494018555 2023-09-05 20:02:30,685 44k INFO ====> Epoch: 9953, cost 14.77 s 2023-09-05 20:02:45,261 44k INFO ====> Epoch: 9954, cost 14.58 s 2023-09-05 20:02:59,849 44k INFO ====> Epoch: 9955, cost 14.59 s 2023-09-05 20:03:14,531 44k INFO ====> Epoch: 9956, cost 14.68 s 2023-09-05 20:03:29,032 44k INFO ====> Epoch: 9957, cost 14.50 s 2023-09-05 20:03:43,559 44k INFO ====> Epoch: 9958, cost 14.53 s 2023-09-05 20:03:57,991 44k INFO ====> Epoch: 9959, cost 14.43 s 2023-09-05 20:04:12,650 44k INFO ====> Epoch: 9960, cost 14.66 s 2023-09-05 20:04:27,134 44k INFO ====> Epoch: 9961, cost 14.48 s 2023-09-05 20:04:40,253 44k INFO Train Epoch: 9962 [86%] 2023-09-05 20:04:40,253 44k INFO Losses: [2.133026599884033, 2.779665946960449, 10.339014053344727, 14.193052291870117, 0.515574038028717], step: 209200, lr: 2.878825119205638e-05, reference_loss: 29.9603328704834 2023-09-05 20:04:41,627 44k INFO ====> Epoch: 9962, cost 14.49 s 2023-09-05 20:04:56,254 44k INFO ====> Epoch: 9963, cost 14.63 s 2023-09-05 20:05:10,823 44k INFO ====> Epoch: 9964, cost 14.57 s 2023-09-05 20:05:24,975 44k INFO ====> Epoch: 9965, cost 14.15 s 2023-09-05 20:05:39,349 44k INFO ====> Epoch: 9966, cost 14.37 s 2023-09-05 20:05:54,104 44k INFO ====> Epoch: 9967, cost 14.75 s 2023-09-05 20:06:08,753 44k INFO ====> Epoch: 9968, cost 14.65 s 2023-09-05 20:06:23,302 44k INFO ====> Epoch: 9969, cost 14.55 s 2023-09-05 20:06:38,010 44k INFO ====> Epoch: 9970, cost 14.71 s 2023-09-05 20:06:52,701 44k INFO ====> Epoch: 9971, cost 14.69 s 2023-09-05 20:06:59,039 44k INFO Train Epoch: 9972 [38%] 2023-09-05 20:06:59,040 44k INFO Losses: [2.1886048316955566, 2.615354537963867, 9.162328720092773, 15.306358337402344, 0.40070652961730957], step: 209400, lr: 2.8752286113059648e-05, reference_loss: 29.67335319519043 2023-09-05 20:07:07,551 44k INFO ====> Epoch: 9972, cost 14.85 s 2023-09-05 20:07:22,032 44k INFO ====> Epoch: 9973, cost 14.48 s 2023-09-05 20:07:36,389 44k INFO ====> Epoch: 9974, cost 14.36 s 2023-09-05 20:07:50,822 44k INFO ====> Epoch: 9975, cost 14.43 s 2023-09-05 20:08:05,605 44k INFO ====> Epoch: 9976, cost 14.78 s 2023-09-05 20:08:20,153 44k INFO ====> Epoch: 9977, cost 14.55 s 2023-09-05 20:08:34,615 44k INFO ====> Epoch: 9978, cost 14.46 s 2023-09-05 20:08:49,338 44k INFO ====> Epoch: 9979, cost 14.72 s 2023-09-05 20:09:03,744 44k INFO ====> Epoch: 9980, cost 14.41 s 2023-09-05 20:09:17,973 44k INFO Train Epoch: 9981 [90%] 2023-09-05 20:09:17,973 44k INFO Losses: [2.201174259185791, 2.566768169403076, 10.217294692993164, 15.669974327087402, 0.6915504932403564], step: 209600, lr: 2.8719955959627092e-05, reference_loss: 31.346763610839844 2023-09-05 20:09:23,429 44k INFO Saving model and optimizer state at iteration 9981 to ./logs\44k\G_209600.pth 2023-09-05 20:09:24,139 44k INFO Saving model and optimizer state at iteration 9981 to ./logs\44k\D_209600.pth 2023-09-05 20:09:25,217 44k INFO .. Free up space by deleting ckpt ./logs\44k\G_207200.pth 2023-09-05 20:09:25,251 44k INFO .. Free up space by deleting ckpt ./logs\44k\D_207200.pth 2023-09-05 20:09:25,705 44k INFO ====> Epoch: 9981, cost 21.96 s 2023-09-05 20:09:39,980 44k INFO ====> Epoch: 9982, cost 14.28 s 2023-09-05 20:09:54,497 44k INFO ====> Epoch: 9983, cost 14.52 s 2023-09-05 20:10:09,182 44k INFO ====> Epoch: 9984, cost 14.69 s 2023-09-05 20:10:23,572 44k INFO ====> Epoch: 9985, cost 14.39 s 2023-09-05 20:10:38,023 44k INFO ====> Epoch: 9986, cost 14.45 s 2023-09-05 20:10:52,745 44k INFO ====> Epoch: 9987, cost 14.72 s 2023-09-05 20:11:07,477 44k INFO ====> Epoch: 9988, cost 14.73 s 2023-09-05 20:11:21,860 44k INFO ====> Epoch: 9989, cost 14.38 s 2023-09-05 20:11:36,274 44k INFO ====> Epoch: 9990, cost 14.41 s 2023-09-05 20:11:43,398 44k INFO Train Epoch: 9991 [43%] 2023-09-05 20:11:43,399 44k INFO Losses: [2.396507740020752, 2.3643980026245117, 9.204129219055176, 14.846781730651855, 0.5255606770515442], step: 209800, lr: 2.868407620166681e-05, reference_loss: 29.337377548217773 2023-09-05 20:11:51,198 44k INFO ====> Epoch: 9991, cost 14.92 s 2023-09-05 20:12:06,020 44k INFO ====> Epoch: 9992, cost 14.82 s 2023-09-05 20:12:20,427 44k INFO ====> Epoch: 9993, cost 14.41 s 2023-09-05 20:12:34,911 44k INFO ====> Epoch: 9994, cost 14.48 s 2023-09-05 20:12:49,309 44k INFO ====> Epoch: 9995, cost 14.40 s 2023-09-05 20:13:03,867 44k INFO ====> Epoch: 9996, cost 14.56 s 2023-09-05 20:13:18,416 44k INFO ====> Epoch: 9997, cost 14.55 s 2023-09-05 20:13:32,918 44k INFO ====> Epoch: 9998, cost 14.50 s 2023-09-05 20:13:47,462 44k INFO ====> Epoch: 9999, cost 14.54 s 2023-09-05 20:14:02,201 44k INFO Train Epoch: 10000 [95%] 2023-09-05 20:14:02,201 44k INFO Losses: [1.8975999355316162, 2.999809980392456, 11.013410568237305, 15.087615013122559, 0.20069949328899384], step: 210000, lr: 2.865182274602769e-05, reference_loss: 31.199134826660156 2023-09-05 20:14:02,465 44k INFO ====> Epoch: 10000, cost 15.00 s