Package: hcpdiff
HCP-Diffusion📘中文说明📘English document📘中文文档IntroductionHCP-Diffusion is a toolbox for Stable Diffusion models based on🤗 Diffusers.
It facilitates flexiable configurations and component support for training, in comparison withwebuiandsd-scripts.This toolbox supportsColossal-AI, which can significantly reduce GPU memory usage.HCP-Diffusion can unify existing training methods for text-to-image generation (e.g., Prompt-tuning, Textual Inversion, DreamArtist, Fine-tuning, DreamBooth, LoRA, ControlNet, etc) and model structures through a single.yamlconfiguration file.The toolbox has also implemented an upgraded version of DreamArtist with LoRA, named DreamArtist++, for one-shot text-to-image generation.
Compared to DreamArtist, DreamArtist++ is more stable with higher image quality and generation controllability, and faster training speed.FeaturesLayer-wise LoRA (with Conv2d)Layer-wise fine-tuningLayer-wise model ensemblePrompt-tuning with multiple wordsDreamArtist and DreamArtist++Aspect Ratio Bucket (ARB) with automatic clusteringMultiple datasets with multiple data sourcesImage attention maskWord attention multiplierCustom words that occupy multiple wordsMaximum sentence length expansion🤗 AccelerateColossal-AIxFormersfor UNet and text-encoderCLIP skipTag shuffle and dropoutSafetensorssupportControlnet(support training)Min-SNR lossCustom optimizer (Lion, DAdaptation, pytorch-optimizer, ...)Custom lr schedulerSDXL supportInstallInstall with pip:pipinstallhcpdiff# Start a new project and make initializationhcpinitInstall from source:gitclonehttps://github.com/7eu7d7/HCP-Diffusion.gitcdHCP-Diffusion
pipinstall-e.# Modified based on this project or start a new project and make initialization## hcpinitTo use xFormers to reduce VRAM usage and accelerate training:# use condacondainstallxformers-cxformers# use pippipinstallxformers>=0.0.17User guidanceTrainingTraining scripts based on 🤗 Accelerate or Colossal-AI are provided.For 🤗 Accelerate, you may need toconfigure the environmentbefore launching the scripts.For Colossal-AI, you can usetorchrunto launch the scripts.# with Accelerateaccelerate launch -m hcpdiff.train_ac --cfg cfgs/train/cfg_file.yaml# with Accelerate and only one GPUaccelerate launch -m hcpdiff.train_ac_single --cfg cfgs/train/cfg_file.yaml# with Colossal-AI# pip install colossalaitorchrun --nproc_per_node 1 -m hcpdiff.train_colo --cfg cfgs/train/cfg_file.yamlInferencepython -m hcpdiff.visualizer --cfg cfgs/infer/cfg.yaml pretrained_model=pretrained_model_path \prompt='positive_prompt' \neg_prompt='negative_prompt' \seed=42Conversion of Stable Diffusion modelsThe framework is based on 🤗 Diffusers. So it needs to convert the original Stable Diffusion model into a supported format using thescripts provided by 🤗 Diffusers.Download theconfig fileConvert models based on config filepython-mhcpdiff.tools.sd2diffusers\--checkpoint_path"path_to_stable_diffusion_model"\--original_config_file"path_to_config_file"\--dump_path"save_directory"\[--extract_ema]# Extract EMA model[--from_safetensors]# Whether the original model is in safetensors format[--to_safetensors]# Whether to save to safetensors formatConvert VAE:python-mhcpdiff.tools.sd2diffusers\--vae_pt_path"path_to_VAE_model"\--original_config_file"path_to_config_file"\--dump_path"save_directory"[--from_safetensors]TutorialsModel Training TutorialDreamArtist++ TutorialModel Inference TutorialConfiguration File Explanationwebui Model Conversion TutorialContributingYou are welcome to contribute more models and features to this toolbox!TeamThis toolbox is maintained byHCP-Lab, SYSU.Citation@article{DBLP:journals/corr/abs-2211-11337,
  author    = {Ziyi Dong and
               Pengxu Wei and
               Liang Lin},
  title     = {DreamArtist: Towards Controllable One-Shot Text-to-Image Generation
               via Positive-Negative Prompt-Tuning},
  journal   = {CoRR},
  volume    = {abs/2211.11337},
  year      = {2022},
  doi       = {10.48550/arXiv.2211.11337},
  eprinttype = {arXiv},
  eprint    = {2211.11337},
}