aaroncaozj/SD1.5_4steps-DMD

SDv1.5 4-step DMD Distillation

4-step distilled SDv1.5 models trained with different DMD variants, one folder per training method.

Experiments were conducted based on DMD2 Codebase. Both models also use the backward simulation specially designed in the DMD2 on SDv1.5 (the original code was only for SDXL).

Folder	Method	real_guidance_scale	Checkpoint	CLIP score
/decoupled-dmd	Decoupled DMD	3.0	checkpoint_model_002000.bin	0.325
/f-divergence-dmd	f-Divergence DMD (JS)	1.75	checkpoint_model_002000.bin	0.331

Both checkpoints were obtained at 2000 steps (2000 generator updates, 10000 guidance updates).

decoupled-dmd

The training method of this distillation model follows Alibaba's Decoupled DMD: CFG Augmentation as the Spear, Distribution Matching as the Shield, which decouples the DMD Loss into a CFG Augmentation Loss under two independent time steps and a new Distribution Matching Loss as regularization.

Training with the following parameters:

--real_guidance_scale 3.0 \
--fake_guidance_scale 1.0 \
--max_grad_norm 10.0 \
--use_fp16 \
--log_loss \
--dfake_gen_update_ratio 5 \
--fsdp \
--denoising \
--num_denoising_step 4 \
--denoising_timestep 1000 \
--backward_simulation \
--use_decoupled_dmd \
--min_step_percent 0.0 \
--max_step_percent 1.0 \

The checkpoint obtained at 2000 steps (with 2000 generator updates, 10000 guidance updates, and real_guidance_scale = 3.0) got a CLIP score of 0.325.

f-divergence-dmd

The training method of this distillation model follows NVIDIA's One-step Diffusion Models with f-Divergence Distribution Matching, using an Jensen-Shannon divergence with coefficients to replace the distribution matching loss calculation method in DMD2 Codebase.

Training with the following parameters:

--real_guidance_scale 1.75 \
--fake_guidance_scale 1.0 \
--max_grad_norm 10.0 \
--use_fp16 \
--dfake_gen_update_ratio 5 \
--fsdp \
--denoising \
--num_denoising_step 4 \
--denoising_timestep 1000 \
--backward_simulation \
--cls_on_clean_image \
--gen_cls_loss \
--gen_cls_loss_weight 5e-3 \
--guidance_cls_loss_weight 1e-2 \
--diffusion_gan \
--diffusion_gan_max_timestep 1000 \
--use_f_divergence \
--divergence_type "JS" \

The checkpoint obtained at 2000 steps (with 2000 generator updates, 10000 guidance updates, and real_guidance_scale = 1.75) achieved a CLIP score of 0.331.

This result slightly outperforms the base model. During experiments with SDv1.5 at various CFG scales, we found that its prompt alignment and response capabilities are inherently limited.

Download

hf download aaroncaozj/SD1.5_4steps-DMD --include "f-divergence-dmd/*" --local-dir ./SD1.5_4steps-DMD

This repo merges the former repos aaroncaozj/SDv1.5-4steps-Decoupled-DMD and aaroncaozj/SDv1.5-4steps-f-divergence-DMD.