aaroncaozj/SDv1.5-4steps-f-divergence-DMD

Method

The training method of this distillation model follows NVIDIA's One-step Diffusion Models with f-Divergence Distribution Matching, using an Jensen-Shannon divergence with coefficients to replace the distribution matching loss calculation method in DMD2 Codebase.

It also uses the backward simulation specially designed in the DMD2 on SDv1.5 (the original code was only for SDXL).

Training with the following parameters:

--real_guidance_scale 1.75 \
--fake_guidance_scale 1.0 \
--max_grad_norm 10.0 \
--use_fp16 \
--dfake_gen_update_ratio 5 \
--fsdp \
--denoising \
--num_denoising_step 4 \
--denoising_timestep 1000 \
--backward_simulation \
--cls_on_clean_image \
--gen_cls_loss \
--gen_cls_loss_weight 5e-3 \
--guidance_cls_loss_weight 1e-2 \
--diffusion_gan \
--diffusion_gan_max_timestep 1000 \
--use_f_divergence \
--divergence_type "JS" \

Result

The checkpoint obtained at 2000 steps (with 2000 generator updates, 10000 guidance updates, and real_guidance_scale = 1.75) achieved a CLIP score of 0.331.

This result slightly outperforms the base model. During experiments with SDv1.5 at various CFG scales, we found that its prompt alignment and response capabilities are inherently limited.