|
12/28/2024 05:07:00 PM [93m Disabling AMP, not recommended.[0m |
|
12/28/2024 05:07:00 PM Seed: 756498848 |
|
12/28/2024 05:07:00 PM unet attention_head_dim: 8 |
|
12/28/2024 05:07:00 PM Inferred yaml: v1-inference.yaml, attn: sd1, prediction_type: epsilon |
|
12/28/2024 05:07:01 PM * Using default (DDPM) noise scheduler for training: ddpm |
|
12/28/2024 05:07:01 PM * Using SDP attention * |
|
12/28/2024 05:07:01 PM * DLMA resolution 512, buckets: [[512, 512], [576, 448], [448, 576], [640, 384], [384, 640], [768, 320], [320, 768], [896, 256], [256, 896], [1024, 256], [256, 1024]] |
|
12/28/2024 05:07:01 PM Preloading images... |
|
12/28/2024 05:07:06 PM * Loaded 40 validation images for validation set 'val' from ../data/val |
|
12/28/2024 05:07:06 PM * DLMA initialized with 40 images. |
|
12/28/2024 05:07:06 PM ** Dataset 'val': 40 batches, num_images: 40, batch_size: 1 |
|
12/28/2024 05:07:06 PM * [91mAspect ratio bucket ('default_batch', 768, 320) has only 1 images[0m. At batch size 1 this makes for an effective multiplier of 2.0, which may cause problems. Consider adding 1 or more images with aspect ratio 12:5, or reducing your batch_size. |
|
12/28/2024 05:07:06 PM - Plugin plugins.interruptible.InterruptiblePlugin loaded to <class 'plugins.interruptible.InterruptiblePlugin'> |
|
12/28/2024 05:07:06 PM * DLMA initialized with 1913 images. |
|
12/28/2024 05:07:06 PM ** Dataset 'train': 1913 batches, num_images: 1913, batch_size: 1 |
|
12/28/2024 05:07:06 PM [36m * unet optimizer: AdamW (686 parameters) *[0m |
|
12/28/2024 05:07:06 PM [36m lr: 3e-07, betas: [0.9, 0.99], epsilon: 1e-08, weight_decay: 0.01 *[0m |
|
12/28/2024 05:07:06 PM Grad scaler enabled: False (amp mode) |
|
12/28/2024 05:07:06 PM Pretraining GPU Memory: 5102 / 24564 MB |
|
12/28/2024 05:07:06 PM saving ckpts every 1000000000.0 minutes |
|
12/28/2024 05:07:06 PM saving ckpts every 4 epochs |
|
12/28/2024 05:07:06 PM unet device: cuda:0, precision: torch.float32, training: True |
|
12/28/2024 05:07:06 PM text_encoder device: cuda:0, precision: torch.float32, training: False |
|
12/28/2024 05:07:06 PM vae device: cuda:0, precision: torch.float32, training: False |
|
12/28/2024 05:07:06 PM scheduler: <class 'diffusers.schedulers.scheduling_ddpm.DDPMScheduler'> |
|
12/28/2024 05:07:06 PM [32mProject name: [0m[92mff7r_4e-5cosine_everything-cont-20-ep50-3e-7_bs1[0m |
|
12/28/2024 05:07:06 PM [32mgrad_accum: [0m[92m1[0m |
|
12/28/2024 05:07:06 PM [32mbatch_size: [0m[92m1[0m |
|
12/28/2024 05:07:06 PM [32mepoch_len: [92m1913[0m |
|
12/28/2024 05:45:02 PM * Saving diffusers model to ../logs/ff7r_4e-5cosine_everything-cont-20-ep50-3e-7_bs1-20241228-170700/ckpts/ff7r_4e-5cosine_everything-cont-20-ep50-3e-7_bs1-ep04-gs07652 |
|
12/28/2024 05:45:04 PM Saving optimizer state to ../logs/ff7r_4e-5cosine_everything-cont-20-ep50-3e-7_bs1-20241228-170700/ckpts/ff7r_4e-5cosine_everything-cont-20-ep50-3e-7_bs1-ep04-gs07652 |
|
12/28/2024 05:45:08 PM [36mTraining complete[0m |
|
12/28/2024 05:45:08 PM Total training time took 38.03 minutes, total steps: 7652 |
|
12/28/2024 05:45:08 PM Average epoch time: 9.46 minutes |
|
12/28/2024 05:45:08 PM [97m ***************************[0m |
|
12/28/2024 05:45:08 PM [97m **** Finished training ****[0m |
|
12/28/2024 05:45:08 PM [97m ***************************[0m |
|
|
