Package: diffusers-unchained
🤗 Diffusers is the go-to library for state-of-the-art pretrained diffusion models for generating images, audio, and even 3D structures of molecules. Whether you're looking for a simple inference solution or training your own diffusion models, 🤗 Diffusers is a modular toolbox that supports both. Our library is designed with a focus onusability over performance,simple over easy, andcustomizability over abstractions.🤗 Diffusers offers three core components:State-of-the-artdiffusion pipelinesthat can be run in inference with just a few lines of code.Interchangeable noiseschedulersfor different diffusion speeds and output quality.Pretrainedmodelsthat can be used as building blocks, and combined with schedulers, for creating your own end-to-end diffusion systems.InstallationWe recommend installing 🤗 Diffusers in a virtual environment from PyPi or Conda. For more details about installingPyTorchandFlax, please refer to their official documentation.PyTorchWithpip(official package):pipinstall--upgradediffusers[torch]Withconda(maintained by the community):condainstall-cconda-forgediffusersFlaxWithpip(official package):pipinstall--upgradediffusers[flax]Apple Silicon (M1/M2) supportPlease refer to theHow to use Stable Diffusion in Apple Siliconguide.QuickstartGenerating outputs is super easy with 🤗 Diffusers. To generate an image from text, use thefrom_pretrainedmethod to load any pretrained diffusion model (browse theHubfor 4000+ checkpoints):fromdiffusersimportDiffusionPipelinepipeline=DiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5")pipeline.to("cuda")pipeline("An image of a squirrel in Picasso style").images[0]You can also dig into the models and schedulers toolbox to build your own diffusion system:fromdiffusersimportDDPMScheduler,UNet2DModelfromPILimportImageimporttorchimportnumpyasnpscheduler=DDPMScheduler.from_pretrained("google/ddpm-cat-256")model=UNet2DModel.from_pretrained("google/ddpm-cat-256").to("cuda")scheduler.set_timesteps(50)sample_size=model.config.sample_sizenoise=torch.randn((1,3,sample_size,sample_size)).to("cuda")input=noisefortinscheduler.timesteps:withtorch.no_grad():noisy_residual=model(input,t).sampleprev_noisy_sample=scheduler.step(noisy_residual,t,input).prev_sampleinput=prev_noisy_sampleimage=(input/2+0.5).clamp(0,1)image=image.cpu().permute(0,2,3,1).numpy()[0]image=Image.fromarray((image*255).round().astype("uint8"))imageCheck out theQuickstartto launch your diffusion journey today!How to navigate the documentationDocumentationWhat can I learn?TutorialA basic crash course for learning how to use the library's most important features like using models and schedulers to build your own diffusion system, and training your own diffusion model.LoadingGuides for how to load and configure all the components (pipelines, models, and schedulers) of the library, as well as how to use different schedulers.Pipelines for inferenceGuides for how to use pipelines for different inference tasks, batched generation, controlling generated outputs and randomness, and how to contribute a pipeline to the library.OptimizationGuides for how to optimize your diffusion model to run faster and consume less memory.TrainingGuides for how to train a diffusion model for different tasks with different training techniques.Supported pipelinesPipelinePaperTasksalt_diffusionAltDiffusionImage-to-Image Text-Guided Generationaudio_diffusionAudio DiffusionUnconditional Audio GenerationcontrolnetControlNet with Stable DiffusionImage-to-Image Text-Guided Generationcycle_diffusionCycle DiffusionImage-to-Image Text-Guided Generationdance_diffusionDance DiffusionUnconditional Audio GenerationddpmDenoising Diffusion Probabilistic ModelsUnconditional Image GenerationddimDenoising Diffusion Implicit ModelsUnconditional Image Generationlatent_diffusionHigh-Resolution Image Synthesis with Latent Diffusion ModelsText-to-Image Generationlatent_diffusionHigh-Resolution Image Synthesis with Latent Diffusion ModelsSuper Resolution Image-to-Imagelatent_diffusion_uncondHigh-Resolution Image Synthesis with Latent Diffusion ModelsUnconditional Image Generationpaint_by_examplePaint by Example: Exemplar-based Image Editing with Diffusion ModelsImage-Guided Image InpaintingpndmPseudo Numerical Methods for Diffusion Models on ManifoldsUnconditional Image Generationscore_sde_veScore-Based Generative Modeling through Stochastic Differential EquationsUnconditional Image Generationscore_sde_vpScore-Based Generative Modeling through Stochastic Differential EquationsUnconditional Image Generationsemantic_stable_diffusionSemantic GuidanceText-Guided Generationstable_diffusion_text2imgStable DiffusionText-to-Image Generationstable_diffusion_img2imgStable DiffusionImage-to-Image Text-Guided Generationstable_diffusion_inpaintStable DiffusionText-Guided Image Inpaintingstable_diffusion_panoramaMultiDiffusionText-to-Panorama Generationstable_diffusion_pix2pixInstructPix2PixText-Guided Image Editingstable_diffusion_pix2pix_zeroZero-shot Image-to-Image TranslationText-Guided Image Editingstable_diffusion_attend_and_exciteAttend and Excite for Stable DiffusionText-to-Image Generationstable_diffusion_self_attention_guidanceSelf-Attention GuidanceText-to-Image Generationstable_diffusion_image_variationStable Diffusion Image VariationsImage-to-Image Generationstable_diffusion_latent_upscaleStable Diffusion Latent UpscalerText-Guided Super Resolution Image-to-Imagestable_diffusion_2Stable Diffusion 2Text-to-Image Generationstable_diffusion_2Stable Diffusion 2Text-Guided Image Inpaintingstable_diffusion_2Depth-Conditional Stable DiffusionDepth-to-Image Generationstable_diffusion_2Stable Diffusion 2Text-Guided Super Resolution Image-to-Imagestable_diffusion_safeSafe Stable DiffusionText-Guided Generationstable_unclipStable unCLIPText-to-Image Generationstable_unclipStable unCLIPImage-to-Image Text-Guided Generationstochastic_karras_veElucidating the Design Space of Diffusion-Based Generative ModelsUnconditional Image GenerationunclipHierarchical Text-Conditional Image Generation with CLIP LatentsText-to-Image Generationversatile_diffusionVersatile Diffusion: Text, Images and Variations All in One Diffusion ModelText-to-Image Generationversatile_diffusionVersatile Diffusion: Text, Images and Variations All in One Diffusion ModelImage Variations Generationversatile_diffusionVersatile Diffusion: Text, Images and Variations All in One Diffusion ModelDual Image and Text Guided Generationvq_diffusionVector Quantized Diffusion Model for Text-to-Image SynthesisText-to-Image GenerationContributionWe ❤️  contributions from the open-source community!
If you want to contribute to this library, please check out ourContribution guide.
You can look out forissuesyou'd like to tackle to contribute to the library.SeeGood first issuesfor general opportunities to contributeSeeNew model/pipelineto contribute exciting new diffusion models / diffusion pipelinesSeeNew schedulerAlso, say 👋 in our public Discord channel. We discuss the hottest trends about diffusion models, help each other with contributions, personal projects or
just hang out ☕.CreditsThis library concretizes previous work by many different authors and would not have been possible without their great research and implementations. We'd like to thank, in particular, the following implementations which have helped us in our development and without which the API could not have been as polished today:@CompVis' latent diffusion models library, availablehere@hojonathanho original DDPM implementation, availablehereas well as the extremely useful translation into PyTorch by @pesser, availablehere@ermongroup's DDIM implementation, availablehere@yang-song's Score-VE and Score-VP implementations, availablehereWe also want to thank @heejkoo for the very helpful overview of papers, code and resources on diffusion models, availablehereas well as @crowsonkb and @rromb for useful discussions and insights.Citation@misc{von-platen-etal-2022-diffusers,author={Patrick von Platen and Suraj Patil and Anton Lozhkov and Pedro Cuenca and Nathan Lambert and Kashif Rasul and Mishig Davaadorj and Thomas Wolf},title={Diffusers: State-of-the-art diffusion models},year={2022},publisher={GitHub},journal={GitHub repository},howpublished={\url{https://github.com/huggingface/diffusers}}}