Package: ppdiffusers
特性|安装|快速开始|模型部署PPDiffusers: Diffusers toolbox implemented based on PaddlePaddlePPDiffusers是一款支持多种模态（如文本图像跨模态、图像、语音）扩散模型（Diffusion Model）训练和推理的国产化工具箱，依托于PaddlePaddle框架和PaddleNLP自然语言处理开发库。News 📢🔥2023.12.12 发布 0.19.4 版本，修复已知的部分 BUG，修复 0D Tensor 的 Warning，新增 SDXL 的 FastdeployPipeline。🔥2023.09.27 发布 0.19.3 版本，新增SDXL，支持Text2Image、Img2Img、Inpainting、InstructPix2Pix等任务，支持DreamBooth Lora训练；
新增UniDiffuser，通过统一的多模态扩散过程支持文生图、图生文等任务；
新增文本条件视频生成模型LVDM，支持训练与推理；
新增文图生成模型Kandinsky 2.2，Consistency models；
Stable Diffusion支持BF16 O2训练，效果对齐FP32；LoRA加载升级，支持加载SDXL的LoRA权重；Controlnet升级，支持ControlNetImg2Img、ControlNetInpaint、StableDiffusionXLControlNet等。🔥2023.06.20 发布 0.16.1 版本，新增T2I-Adapter，支持训练与推理；ControlNet升级，支持reference only推理；新增WebUIStableDiffusionPipeline，
支持通过prompt的方式动态加载lora、textual_inversion权重；
新增StableDiffusionHiresFixPipeline，支持高分辨率修复；
新增关键点控制生成任务评价指标COCOeval；
新增多种模态扩散模型Pipeline，包括视频生成（Text-to-Video-Synth、Text-to-Video-Zero）、音频生成（AudioLDM、Spectrogram Diffusion）；新增文图生成模型IF。🔥2023.03.29 发布 0.14.0 版本，新增LoRA、ControlNet，支持训练与推理；
模型加载升级，可直接加载HF Diffusers的权重（safetensors和pt）或SD等原库的Lightning权重进行推理，支持加载Civitai社区的LoRA权重；支持xformers训练与推理；
新增用于超高分辨率生成的VAE tiling；
新增Instruct Pix2Pix、Semantic guidance、Depth2image等模型。特性📦 SOTA扩散模型Pipelines集合我们提供SOTA（State-of-the-Art）的扩散模型Pipelines集合。
目前PPDiffusers已经集成了100+Pipelines，支持文图生成（Text-to-Image Generation）、文本引导的图像编辑（Text-Guided Image Inpainting）、文本引导的图像变换（Image-to-Image Text-Guided Generation）、文本条件的视频生成（Text-to-Video Generation）、超分（Super Superresolution）、文本条件的音频生成（Text-to-Audio Generation）在内的10余项任务，覆盖文本、图像、视频、音频等多种模态。
如果想要了解当前支持的所有Pipelines以及对应的来源信息，可以阅读🔥 PPDiffusers Pipelines文档。🔊 提供丰富的Noise Scheduler我们提供了丰富的噪声调度器（Noise Scheduler），可以对速度与质量进行权衡，用户可在推理时根据需求快速切换使用。
当前PPDiffusers已经集成了14+Scheduler，不仅支持DDPM、DDIM和PNDM，还支持最新的🔥 DPMSolver！🎛️ 提供多种扩散模型组件我们提供了多种扩散模型组件，如UNet1DModel、UNet2DModel、UNet2DConditionModel、UNet3DConditionModel、VQModel、AutoencoderKL等。📖 提供丰富的训练和推理教程我们提供了丰富的训练教程，不仅支持扩散模型的二次开发微调，如基于Textual Inversion和DreamBooth使用3-5张图定制化训练生成图像的风格或物体，还支持🔥 Latent Diffusion Model、🔥 ControlNet、🔥 T2I-Adapter等扩散模型的训练！
此外，我们还提供了丰富的🔥 Pipelines推理样例。🚀 支持FastDeploy高性能部署我们提供基于FastDeploy的🔥 高性能Stable Diffusion Pipeline，更多有关FastDeploy进行多推理引擎后端高性能部署的信息请参考🔥 高性能FastDeploy推理教程。安装环境依赖pip install -r requirements.txt关于PaddlePaddle安装的详细教程请查看Installation。pip安装pipinstall--upgradeppdiffusers手动安装gitclonehttps://github.com/PaddlePaddle/PaddleMIXcdPaddleMIX/ppdiffusers
pythonsetup.pyinstall快速开始我们将以扩散模型的典型代表Stable Diffusion为例，带你快速了解PPDiffusers。Stable Diffusion基于潜在扩散模型（Latent Diffusion Models），专门用于文图生成（Text-to-Image Generation）任务。该模型是由来自CompVis,Stability AI,LAION以及RunwayML的工程师共同开发完成，目前发布了v1和v2两个版本。v1版本采用了LAION-5B数据集子集（分辨率为 512x512）进行训练，并具有以下架构设置：自动编码器下采样因子为8，UNet大小为860M，文本编码器为CLIP ViT-L/14。v2版本相较于v1版本在生成图像的质量和分辨率等进行了改善。Stable Diffusion重点模型权重Stable Diffusion 模型支持的权重（英文）我们只需要将下面的"xxxx"，替换成所需的权重名，即可快速使用！fromppdiffusersimport*pipe_text2img=StableDiffusionPipeline.from_pretrained("xxxx")pipe_img2img=StableDiffusionImg2ImgPipeline.from_pretrained("xxxx")pipe_inpaint_legacy=StableDiffusionInpaintPipelineLegacy.from_pretrained("xxxx")pipe_mega=StableDiffusionMegaPipeline.from_pretrained("xxxx")# pipe_mega.text2img() 等于 pipe_text2img()# pipe_mega.img2img() 等于 pipe_img2img()# pipe_mega.inpaint_legacy() 等于 pipe_inpaint_legacy()PPDiffusers支持的模型名称支持加载的Pipeline备注huggingface.co地址CompVis/stable-diffusion-v1-4StableDiffusionPipeline、StableDiffusionImg2ImgPipeline、StableDiffusionInpaintPipelineLegacy、StableDiffusionMegaPipeline、StableDiffusionPipelineAllinOneStable-Diffusion-v1-4 使用 Stable-Diffusion-v1-2 的权重进行初始化。随后在"laion-aesthetics v2 5+"数据集上以512x512分辨率微调了225k步数，对文本使用了10%的dropout（即：训练过程中文图对中的文本有 10% 的概率会变成空文本）。模型使用了CLIP ViT-L/14作为文本编码器。地址CompVis/ldm-text2im-large-256LDMTextToImagePipelineLDM论文LDM-KL-8-G* 权重。地址CompVis/ldm-super-resolution-4x-openimagesLDMSuperResolutionPipelineLDM论文LDM-VQ-4 权重，原始权重链接。地址runwayml/stable-diffusion-v1-5StableDiffusionPipeline、StableDiffusionImg2ImgPipeline、StableDiffusionInpaintPipelineLegacy、StableDiffusionMegaPipeline、StableDiffusionPipelineAllinOneStable-Diffusion-v1-5 使用 Stable-Diffusion-v1-2 的权重进行初始化。随后在"laion-aesthetics v2 5+"数据集上以512x512分辨率微调了595k步数，对文本使用了10%的dropout（即：训练过程中文图对中的文本有 10% 的概率会变成空文本）。模型同样也使用了CLIP ViT-L/14作为文本编码器。地址runwayml/stable-diffusion-inpaintingStableDiffusionInpaintPipelineStable-Diffusion-Inpainting 使用 Stable-Diffusion-v1-2 的权重进行初始化。首先进行了595k步的常规训练（实际也就是 Stable-Diffusion-v1-5 的权重），然后进行了440k步的 inpainting 修复训练。对于 inpainting 修复训练，给 UNet 额外增加了5输入通道（其中4个用于被 Mask 遮盖住的图片，1个用于 Mask 本身）。在训练期间，会随机生成 Mask，并有25%概率会将原始图片全部 Mask 掉。地址stabilityai/stable-diffusion-2-baseStableDiffusionPipeline、StableDiffusionImg2ImgPipeline、StableDiffusionInpaintPipelineLegacy、StableDiffusionMegaPipeline、StableDiffusionPipelineAllinOne该模型首先在LAION-5B 256x256 子集上（过滤条件：punsafe = 0.1 的 LAION-NSFW 分类器和 审美分数大于等于 4.5 ）从头开始训练550k步，然后又在分辨率>= 512x512的同一数据集上进一步训练850k步。地址stabilityai/stable-diffusion-2StableDiffusionPipeline、StableDiffusionImg2ImgPipeline、StableDiffusionInpaintPipelineLegacy、StableDiffusionMegaPipeline、StableDiffusionPipelineAllinOnestable-diffusion-2 使用 stable-diffusion-2-base 权重进行初始化，首先在同一数据集上（512x512分辨率）使用v-objective训练了150k步。然后又在768x768分辨率上使用v-objective继续训练了140k步。地址stabilityai/stable-diffusion-2-inpaintingStableDiffusionInpaintPipelinestable-diffusion-2-inpainting 使用 stable-diffusion-2-base 权重初始化，并且额外训练了200k步。训练过程使用了LAMA中提出的 Mask 生成策略，并且使用 Mask 图片的 Latent 表示（经过 VAE 编码）作为附加条件。地址stabilityai/stable-diffusion-x4-upscalerStableDiffusionUpscalePipeline该模型在LAION 10M子集上（>2048x2048）训练了 1.25M 步。该模型还在分辨率为512x512的图像上使用Text-guided Latent Upscaling Diffusion Model进行了训练。除了文本输入之外，它还接收noise_level作为输入参数，因此我们可以使用预定义的 Scheduler向低分辨率的输入图片添加噪声。地址hakurei/waifu-diffusionStableDiffusionPipeline、StableDiffusionImg2ImgPipeline、StableDiffusionInpaintPipelineLegacy、StableDiffusionMegaPipeline、StableDiffusionPipelineAllinOnewaifu-diffusion-v1-2 使用 stable-diffusion-v1-4 权重初始化，并且在高质量动漫图像数据集上进行微调后得到的模型。用于微调的数据是680k文本图像样本，这些样本是通过booru 网站下载的。地址hakurei/waifu-diffusion-v1-3StableDiffusionPipeline、StableDiffusionImg2ImgPipeline、StableDiffusionInpaintPipelineLegacy、StableDiffusionMegaPipeline、StableDiffusionPipelineAllinOnewaifu-diffusion-v1-3 是 waifu-diffusion-v1-2 基础上进一步训练得到的。他们对数据集进行了额外操作：（1）删除下划线；（2）删除括号；（3）用逗号分隔每个booru 标签；（4）随机化标签顺序。地址naclbit/trinart_stable_diffusion_v2_60kStableDiffusionPipeline、StableDiffusionImg2ImgPipeline、StableDiffusionInpaintPipelineLegacy、StableDiffusionMegaPipeline、StableDiffusionPipelineAllinOnetrinart_stable_diffusion 使用 stable-diffusion-v1-4 权重初始化，在 40k高分辨率漫画/动漫风格的图片数据集上微调了 8 个 epoch。V2 版模型使用dropouts、10k+ 图像和新的标记策略训练了更长时间。地址naclbit/trinart_stable_diffusion_v2_95kStableDiffusionPipeline、StableDiffusionImg2ImgPipeline、StableDiffusionInpaintPipelineLegacy、StableDiffusionMegaPipeline、StableDiffusionPipelineAllinOne95k步数的结果，其他同上。地址naclbit/trinart_stable_diffusion_v2_115kStableDiffusionPipeline、StableDiffusionImg2ImgPipeline、StableDiffusionInpaintPipelineLegacy、StableDiffusionMegaPipeline、StableDiffusionPipelineAllinOne115k步数的结果，其他同上。地址Deltaadams/Hentai-DiffusionStableDiffusionPipeline、StableDiffusionImg2ImgPipeline、StableDiffusionInpaintPipelineLegacy、StableDiffusionMegaPipeline、StableDiffusionPipelineAllinOneNone地址ringhyacinth/nail-set-diffuserStableDiffusionPipeline、StableDiffusionImg2ImgPipeline、StableDiffusionInpaintPipelineLegacy、StableDiffusionMegaPipeline、StableDiffusionPipelineAllinOne美甲领域的扩散模型，训练数据使用了Weekend地址Linaqruf/anything-v3.0StableDiffusionPipeline、StableDiffusionImg2ImgPipeline、StableDiffusionInpaintPipelineLegacy、StableDiffusionMegaPipeline、StableDiffusionPipelineAllinOne该模型可通过输入几个文本提示词就能生成高质量、高度详细的动漫风格图片，该模型支持使用danbooru 标签文本生成图像。地址Stable Diffusion 模型支持的权重（中文和多语言）PPDiffusers支持的模型名称支持加载的Pipeline备注huggingface.co地址BAAI/AltDiffusionAltDiffusionPipeline、AltDiffusionImg2ImgPipeline该模型使用AltCLIP作为文本编码器，在 Stable Diffusion 基础上训练了双语Diffusion模型，其中训练数据来自WuDao数据集和LAION。地址BAAI/AltDiffusion-m9AltDiffusionPipeline、AltDiffusionImg2ImgPipeline该模型使用9种语言的AltCLIP-m9作为文本编码器，其他同上。地址IDEA-CCNL/Taiyi-Stable-Diffusion-1B-Chinese-v0.1StableDiffusionPipeline、StableDiffusionImg2ImgPipeline、StableDiffusionInpaintPipelineLegacy、StableDiffusionMegaPipeline、StableDiffusionPipelineAllinOne他们将Noah-Wukong数据集 (100M) 和Zero数据集 (23M) 用作预训练的数据集，先用IDEA-CCNL/Taiyi-CLIP-RoBERTa-102M-ViT-L-Chinese对这两个数据集的图文对相似性进行打分，取 CLIP Score 大于 0.2 的图文对作为训练集。 他们使用IDEA-CCNL/Taiyi-CLIP-RoBERTa-102M-ViT-L-Chinese作为初始化的text encoder，冻住stable-diffusion-v1-4(论文) 模型的其他部分，只训练 text encoder，以便保留原始模型的生成能力且实现中文概念的对齐。该模型目前在0.2亿图文对上训练了一个 epoch。 在 32 x A100 上训练了大约100小时，该版本只是一个初步的版本。地址IDEA-CCNL/Taiyi-Stable-Diffusion-1B-Chinese-EN-v0.1StableDiffusionPipeline、StableDiffusionImg2ImgPipeline、StableDiffusionInpaintPipelineLegacy、StableDiffusionMegaPipeline、StableDiffusionPipelineAllinOne他们将Noah-Wukong数据集 (100M) 和Zero数据集 (23M) 用作预训练的数据集，先用IDEA-CCNL/Taiyi-CLIP-RoBERTa-102M-ViT-L-Chinese对这两个数据集的图文对相似性进行打分，取 CLIP Score 大于 0.2 的图文对作为训练集。 他们使用stable-diffusion-v1-4(论文) 模型进行继续训练，其中训练分为两个stage。第一个stage中冻住模型的其他部分，只训练 text encoder ，以便保留原始模型的生成能力且实现中文概念的对齐。第二个stage中将全部模型解冻，一起训练 text encoder 和 diffusion model ，以便 diffusion model 更好的适配中文引导。第一个 stage 他们训练了 80 小时，第二个 stage 训练了 100 小时，两个stage都是用了8 x A100，该版本是一个初步的版本。地址加载HF Diffusers权重fromppdiffusersimportStableDiffusionPipeline# 设置from_hf_hub为True，表示从huggingface hub下载，from_diffusers为True表示加载的是diffusers版Pytorch权重pipe=StableDiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-2",from_hf_hub=True,from_diffusers=True)加载原库的Lightning权重fromppdiffusersimportStableDiffusionPipeline# 可输入网址 或 本地ckpt、safetensors文件pipe=StableDiffusionPipeline.from_pretrained_original_ckpt("https://paddlenlp.bj.bcebos.com/models/community/junnyu/develop/ppdiffusers/chilloutmix_NiPrunedFp32Fix.safetensors")加载HF LoRA权重fromppdiffusersimportDiffusionPipelinepipe=DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0",paddle_dtype=paddle.float16)pipe.load_lora_weights("stabilityai/stable-diffusion-xl-base-1.0",weight_name="sd_xl_offset_example-lora_1.0.safetensors",from_diffusers=True)加载Civitai社区的LoRA权重fromppdiffusersimportStableDiffusionPipelinepipe=StableDiffusionPipeline.from_pretrained("TASUKU2023/Chilloutmix")# 加载lora权重pipe.apply_lora("https://paddlenlp.bj.bcebos.com/models/community/junnyu/develop/ppdiffusers/Moxin_10.safetensors")XFormers加速为了使用XFormers加速，我们需要安装develop版本的paddle，Linux系统的安装命令如下：python-mpipinstallpaddlepaddle-gpu==0.0.0.post117-fhttps://www.paddlepaddle.org.cn/whl/linux/gpu/develop.htmlimportpaddlefromppdiffusersimportStableDiffusionPipelinepipe=StableDiffusionPipeline.from_pretrained("TASUKU2023/Chilloutmix",paddle_dtype=paddle.float16)# 开启xformers加速 默认选择"cutlass"加速pipe.enable_xformers_memory_efficient_attention()# flash 需要使用 A100、A10、3060、3070、3080、3090 等以上显卡。# pipe.enable_xformers_memory_efficient_attention("flash")ToME + ControlNet# 安装develop的ppdiffusers# pip install "ppdiffusers>=0.16.1"importpaddlefromppdiffusersimportControlNetModel,StableDiffusionControlNetPipelinefromppdiffusers.utilsimportload_imagecontrolnet=ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-canny")pipe=StableDiffusionControlNetPipeline.from_pretrained("runwayml/stable-diffusion-v1-5",safety_checker=None,controlnet=controlnet,paddle_dtype=paddle.float16)# Apply ToMe with a 50% merging ratiopipe.apply_tome(ratio=0.5)# Can also use pipe.unet in place of pipe here# 我们可以开启 xformers# pipe.enable_xformers_memory_efficient_attention()generator=paddle.Generator().manual_seed(0)prompt="bird"image=load_image("https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/sd_controlnet/bird_canny.png")image=pipe(prompt,image,generator=generator).images[0]image.save("bird.png")文图生成 （Text-to-Image Generation）importpaddlefromppdiffusersimportStableDiffusionPipelinepipe=StableDiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-2")# 设置随机种子，我们可以复现下面的结果！paddle.seed(5232132133)prompt="a portrait of shiba inu with a red cap growing on its head. intricate. lifelike. soft light. sony a 7 r iv 5 5 mm. cinematic post - processing "image=pipe(prompt,guidance_scale=7.5,height=768,width=768).images[0]image.save("shiba_dog_with_a_red_cap.png")文本引导的图像变换（Image-to-Image Text-Guided Generation）Image-to-Image Text-Guided Generation DemoimportpaddlefromppdiffusersimportStableDiffusionImg2ImgPipelinefromppdiffusers.utilsimportload_imagepipe=StableDiffusionImg2ImgPipeline.from_pretrained("Linaqruf/anything-v3.0",safety_checker=None)url="https://paddlenlp.bj.bcebos.com/models/community/CompVis/data/image_Kurisu.png"image=load_image(url).resize((512,768))# 设置随机种子，我们可以复现下面的结果！paddle.seed(42)prompt="Kurisu Makise, looking at viewer, long hair, standing, 1girl, hair ornament, hair flower, cute, jacket, white flower, white dress"negative_prompt="lowres, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry"image=pipe(prompt=prompt,negative_prompt=negative_prompt,image=image,strength=0.75,guidance_scale=7.5).images[0]image.save("image_Kurisu_img2img.png")文本引导的图像编辑（Text-Guided Image Inpainting）注意！当前有两种版本的图像编辑代码，一个是Legacy版本，一个是正式版本，下面将分别介绍两种代码如何使用！Legacy版本代码importpaddlefromppdiffusersimportStableDiffusionInpaintPipelineLegacyfromppdiffusers.utilsimportload_image# 可选模型权重# CompVis/stable-diffusion-v1-4# runwayml/stable-diffusion-v1-5# stabilityai/stable-diffusion-2-base （原始策略 512x512）# stabilityai/stable-diffusion-2 （v-objective 768x768）# Linaqruf/anything-v3.0# ......img_url="https://paddlenlp.bj.bcebos.com/models/community/CompVis/stable-diffusion-v1-4/overture-creations.png"mask_url="https://paddlenlp.bj.bcebos.com/models/community/CompVis/stable-diffusion-v1-4/overture-creations-mask.png"image=load_image(img_url).resize((512,512))mask_image=load_image(mask_url).resize((512,512))pipe=StableDiffusionInpaintPipelineLegacy.from_pretrained("stabilityai/stable-diffusion-2-base",safety_checker=None)# 设置随机种子，我们可以复现下面的结果！paddle.seed(10245)prompt="a red cat sitting on a bench"image=pipe(prompt=prompt,image=image,mask_image=mask_image,strength=0.75).images[0]image.save("a_red_cat_legacy.png")正式版本代码Tips: 下面的使用方法是新版本的代码，也是官方推荐的代码，注意必须配合runwayml/stable-diffusion-inpainting和stabilityai/stable-diffusion-2-inpainting才可正常使用。importpaddlefromppdiffusersimportStableDiffusionInpaintPipelinefromppdiffusers.utilsimportload_image# 可选模型权重# runwayml/stable-diffusion-inpainting# stabilityai/stable-diffusion-2-inpaintingimg_url="https://paddlenlp.bj.bcebos.com/models/community/CompVis/stable-diffusion-v1-4/overture-creations.png"mask_url="https://paddlenlp.bj.bcebos.com/models/community/CompVis/stable-diffusion-v1-4/overture-creations-mask.png"image=load_image(img_url).resize((512,512))mask_image=load_image(mask_url).resize((512,512))pipe=StableDiffusionInpaintPipeline.from_pretrained("stabilityai/stable-diffusion-2-inpainting")# 设置随机种子，我们可以复现下面的结果！paddle.seed(1024)prompt="Face of a yellow cat, high resolution, sitting on a park bench"image=pipe(prompt=prompt,image=image,mask_image=mask_image).images[0]image.save("a_yellow_cat.png")文本引导的图像放大 & 超分（Text-Guided Image Upscaling & Super-Resolution）Text-Guided Image Upscaling DemoimportpaddlefromppdiffusersimportStableDiffusionUpscalePipelinefromppdiffusers.utilsimportload_imagepipe=StableDiffusionUpscalePipeline.from_pretrained("stabilityai/stable-diffusion-x4-upscaler")url="https://paddlenlp.bj.bcebos.com/models/community/CompVis/data/low_res_cat.png"# 我们人工将原始图片缩小成 128x128 分辨率，最终保存的图片会放大4倍！low_res_img=load_image(url).resize((128,128))prompt="a white cat"image=pipe(prompt=prompt,image=low_res_img).images[0]image.save("upscaled_white_cat.png")Super-Resolution DemoimportpaddlefromppdiffusersimportLDMSuperResolutionPipelinefromppdiffusers.utilsimportload_imagepipe=LDMSuperResolutionPipeline.from_pretrained("CompVis/ldm-super-resolution-4x-openimages")url="https://paddlenlp.bj.bcebos.com/models/community/CompVis/stable-diffusion-v1-4/overture-creations.png"# 我们人工将原始图片缩小成 128x128 分辨率，最终保存的图片会放大4倍！low_res_img=load_image(url).resize((128,128))image=pipe(image=low_res_img,num_inference_steps=100).images[0]image.save("ldm-super-resolution-image.png")模型推理部署除了Paddle动态图运行之外，很多模型还支持将模型导出并使用推理引擎运行。我们提供基于FastDeploy上的StableDiffusion模型部署示例，涵盖文生图、图生图、图像编辑等任务，用户可以按照我们提供StableDiffusion模型导出教程将模型导出，然后使用FastDeployStableDiffusionMegaPipeline进行高性能推理部署！已预先导出的FastDeploy版Stable Diffusion权重注意：当前导出的vae encoder带有随机因素！CompVis/stable-diffusion-v1-4@fastdeployrunwayml/stable-diffusion-v1-5@fastdeployrunwayml/stable-diffusion-inpainting@fastdeploystabilityai/stable-diffusion-2-base@fastdeploystabilityai/stable-diffusion-2@fastdeploystabilityai/stable-diffusion-2-inpainting@fastdeployLinaqruf/anything-v3.0@fastdeployhakurei/waifu-diffusion-v1-3@fastdeployFastDeploy DemoimportpaddleimportfastdeployasfdfromppdiffusersimportFastDeployStableDiffusionMegaPipelinefromppdiffusers.utilsimportload_imagedefcreate_runtime_option(device_id=0,backend="paddle",use_cuda_stream=True):option=fd.RuntimeOption()ifbackend=="paddle":option.use_paddle_backend()else:option.use_ort_backend()ifdevice_id==-1:option.use_cpu()else:option.use_gpu(device_id)ifuse_cuda_stream:paddle_stream=paddle.device.cuda.current_stream(device_id).cuda_streamoption.set_external_raw_stream(paddle_stream)returnoptionruntime_options={"text_encoder":create_runtime_option(0,"paddle"),# use gpu:0"vae_encoder":create_runtime_option(0,"paddle"),# use gpu:0"vae_decoder":create_runtime_option(0,"paddle"),# use gpu:0"unet":create_runtime_option(0,"paddle"),# use gpu:0}fd_pipe=FastDeployStableDiffusionMegaPipeline.from_pretrained("Linaqruf/anything-v3.0@fastdeploy",runtime_options=runtime_options)# text2imgprompt="a portrait of shiba inu with a red cap growing on its head. intricate. lifelike. soft light. sony a 7 r iv 5 5 mm. cinematic post - processing "image_text2img=fd_pipe.text2img(prompt=prompt,num_inference_steps=50).images[0]image_text2img.save("image_text2img.png")# img2imgurl="https://paddlenlp.bj.bcebos.com/models/community/CompVis/data/image_Kurisu.png"image=load_image(url).resize((512,512))prompt="Kurisu Makise, looking at viewer, long hair, standing, 1girl, hair ornament, hair flower, cute, jacket, white flower, white dress"negative_prompt="lowres, bad anatomy, bad hands, text, error, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality, normal quality, jpeg artifacts, signature, watermark, username, blurry"image_img2img=fd_pipe.img2img(prompt=prompt,negative_prompt=negative_prompt,image=image,strength=0.75,guidance_scale=7.5).images[0]image_img2img.save("image_img2img.png")# inpaint_legacyimg_url="https://paddlenlp.bj.bcebos.com/models/community/CompVis/stable-diffusion-v1-4/overture-creations.png"mask_url="https://paddlenlp.bj.bcebos.com/models/community/CompVis/stable-diffusion-v1-4/overture-creations-mask.png"image=load_image(img_url).resize((512,512))mask_image=load_image(mask_url).resize((512,512))prompt="a red cat sitting on a bench"image_inpaint_legacy=fd_pipe.inpaint_legacy(prompt=prompt,image=image,mask_image=mask_image,strength=0.75,num_inference_steps=50).images[0]image_inpaint_legacy.save("image_inpaint_legacy.png")更多任务分类展示文本图像多模文图生成（Text-to-Image Generation）text_to_image_generation-stable_diffusionfromppdiffusersimportStableDiffusionPipeline# 加载模型和schedulerpipe=StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5")# 执行pipeline进行推理prompt="a photo of an astronaut riding a horse on mars"image=pipe(prompt).images[0]# 保存图片image.save("astronaut_rides_horse_sd.png")text_to_image_generation-stable_diffusion_xlimportpaddlefromppdiffusersimportStableDiffusionXLPipelinepipe=StableDiffusionXLPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0",paddle_dtype=paddle.float16,variant="fp16")prompt="a photo of an astronaut riding a horse on mars"generator=paddle.Generator().manual_seed(42)image=pipe(prompt=prompt,generator=generator,num_inference_steps=50).images[0]image.save('sdxl_text2image.png')text_to_image_generation-sdxl_base_with_refinerfromppdiffusersimportDiffusionPipelineimportpaddle# load both base & refinerbase=DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0",)refiner=DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-refiner-1.0",text_encoder_2=base.text_encoder_2,vae=base.vae,paddle_dtype=paddle.float16,variant="fp16",)# Define how many steps and what % of steps to be run on each experts (80/20) heren_steps=40high_noise_frac=0.8prompt="A majestic lion jumping from a big stone at night"prompt="a photo of an astronaut riding a horse on mars"generator=paddle.Generator().manual_seed(42)# run both expertsimage=base(prompt=prompt,output_type="latent",generator=generator,).imagesimage=refiner(prompt=prompt,image=image,generator=generator,).images[0]image.save('text_to_image_generation-sdxl-base-with-refiner-result.png')text_to_image_generation-kandinsky2_2fromppdiffusersimportKandinskyV22Pipeline,KandinskyV22PriorPipelinepipe_prior=KandinskyV22PriorPipeline.from_pretrained("kandinsky-community/kandinsky-2-2-prior")prompt="red cat, 4k photo"out=pipe_prior(prompt)image_emb=out.image_embedszero_image_emb=out.negative_image_embedspipe=KandinskyV22Pipeline.from_pretrained("kandinsky-community/kandinsky-2-2-decoder")image=pipe(image_embeds=image_emb,negative_image_embeds=zero_image_emb,height=768,width=768,num_inference_steps=50,).imagesimage[0].save("text_to_image_generation-kandinsky2_2-result-cat.png")text_to_image_generation-unidiffuserimportpaddlefrompaddlenlp.trainerimportset_seedfromppdiffusersimportUniDiffuserPipelinemodel_id_or_path="thu-ml/unidiffuser-v1"pipe=UniDiffuserPipeline.from_pretrained(model_id_or_path,paddle_dtype=paddle.float16)set_seed(42)# Text variation can be performed with a text-to-image generation followed by a image-to-text generation:# 1. Text-to-image generationprompt="an elephant under the sea"sample=pipe(prompt=prompt,num_inference_steps=20,guidance_scale=8.0)t2i_image=sample.images[0]t2i_image.save("t2i_image.png")text_to_image_generation-deepfloyd_ifimportpaddlefromppdiffusersimportDiffusionPipeline,IFPipeline,IFSuperResolutionPipelinefromppdiffusers.utilsimportpd_to_pil# Stage 1: generate imagespipe=IFPipeline.from_pretrained("DeepFloyd/IF-I-XL-v1.0",variant="fp16",paddle_dtype=paddle.float16)pipe.enable_xformers_memory_efficient_attention()prompt='a photo of a kangaroo wearing an orange hoodie and blue sunglasses standing in front of the eiffel tower holding a sign that says "very deep learning"'prompt_embeds,negative_embeds=pipe.encode_prompt(prompt)image=pipe(prompt_embeds=prompt_embeds,negative_prompt_embeds=negative_embeds,output_type="pd",).images# save intermediate imagepil_image=pd_to_pil(image)pil_image[0].save("text_to_image_generation-deepfloyd_if-result-if_stage_I.png")# save gpu memorypipe.to(paddle_device="cpu")# Stage 2: super resolution stage1super_res_1_pipe=IFSuperResolutionPipeline.from_pretrained("DeepFloyd/IF-II-L-v1.0",text_encoder=None,variant="fp16",paddle_dtype=paddle.float16)super_res_1_pipe.enable_xformers_memory_efficient_attention()image=super_res_1_pipe(image=image,prompt_embeds=prompt_embeds,negative_prompt_embeds=negative_embeds,output_type="pd",).images# save intermediate imagepil_image=pd_to_pil(image)pil_image[0].save("text_to_image_generation-deepfloyd_if-result-if_stage_II.png")# save gpu memorysuper_res_1_pipe.to(paddle_device="cpu")if_stage_Iif_stage_II文本引导的图像放大（Text-Guided Image Upscaling）text_guided_image_upscaling-stable_diffusion_2fromppdiffusersimportStableDiffusionUpscalePipelinefromppdiffusers.utilsimportload_imagepipe=StableDiffusionUpscalePipeline.from_pretrained("stabilityai/stable-diffusion-x4-upscaler")url="https://paddlenlp.bj.bcebos.com/models/community/CompVis/data/low_res_cat.png"low_res_img=load_image(url).resize((128,128))prompt="a white cat"upscaled_image=pipe(prompt=prompt,image=low_res_img).images[0]upscaled_image.save("upsampled_cat_sd2.png")原图像生成图像文本引导的图像编辑（Text-Guided Image Inpainting）text_guided_image_inpainting-stable_diffusion_2importpaddlefromppdiffusersimportPaintByExamplePipelinefromppdiffusers.utilsimportload_imageimg_url="https://paddlenlp.bj.bcebos.com/models/community/Fantasy-Studio/data/image_example_1.png"mask_url="https://paddlenlp.bj.bcebos.com/models/community/Fantasy-Studio/data/mask_example_1.png"example_url="https://paddlenlp.bj.bcebos.com/models/community/Fantasy-Studio/data/reference_example_1.jpeg"init_image=load_image(img_url).resize((512,512))mask_image=load_image(mask_url).resize((512,512))example_image=load_image(example_url).resize((512,512))pipe=PaintByExamplePipeline.from_pretrained("Fantasy-Studio/Paint-by-Example")# 使用fp16加快生成速度withpaddle.amp.auto_cast(True):image=pipe(image=init_image,mask_image=mask_image,example_image=example_image).images[0]image.save("image_guided_image_inpainting-paint_by_example-result.png")原图像掩码图像参考图像生成图像文本引导的图像变换（Image-to-Image Text-Guided Generation）text_guided_image_inpainting-kandinsky2_2importnumpyasnpimportpaddlefromppdiffusersimportKandinskyV22InpaintPipeline,KandinskyV22PriorPipelinefromppdiffusers.utilsimportload_imagepipe_prior=KandinskyV22PriorPipeline.from_pretrained("kandinsky-community/kandinsky-2-2-prior",paddle_dtype=paddle.float16)prompt="a hat"image_emb,zero_image_emb=pipe_prior(prompt,return_dict=False)pipe=KandinskyV22InpaintPipeline.from_pretrained("kandinsky-community/kandinsky-2-2-decoder-inpaint",paddle_dtype=paddle.float16)init_image=load_image("https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/kandinsky/cat.png")mask=np.zeros((768,768),dtype=np.float32)mask[:250,250:-250]=1out=pipe(image=init_image,mask_image=mask,image_embeds=image_emb,negative_image_embeds=zero_image_emb,height=768,width=768,num_inference_steps=50,)image=out.images[0]image.save("text_guided_image_inpainting-kandinsky2_2-result-cat_with_hat.png")原图像生成图像image_to_image_text_guided_generation-stable_diffusionimportpaddlefromppdiffusersimportStableDiffusionImg2ImgPipelinefromppdiffusers.utilsimportload_image# 加载pipelinepipe=StableDiffusionImg2ImgPipeline.from_pretrained("runwayml/stable-diffusion-v1-5")# 下载初始图片url="https://paddlenlp.bj.bcebos.com/models/community/CompVis/stable-diffusion-v1-4/sketch-mountains-input.png"init_image=load_image(url).resize((768,512))prompt="A fantasy landscape, trending on artstation"# 使用fp16加快生成速度withpaddle.amp.auto_cast(True):image=pipe(prompt=prompt,image=init_image,strength=0.75,guidance_scale=7.5).images[0]image.save("fantasy_landscape.png")原图像生成图像image_to_image_text_guided_generation-stable_diffusion_xlimportpaddlefromppdiffusersimportStableDiffusionXLImg2ImgPipelinefromppdiffusers.utilsimportload_imagepipe=StableDiffusionXLImg2ImgPipeline.from_pretrained("stabilityai/stable-diffusion-xl-refiner-1.0",paddle_dtype=paddle.float16,# from_hf_hub=True,# from_diffusers=True,variant="fp16")url="https://paddlenlp.bj.bcebos.com/models/community/westfish/develop-0-19-3/000000009.png"init_image=load_image(url).convert("RGB")prompt="a photo of an astronaut riding a horse on mars"image=pipe(prompt,image=init_image).images[0]image.save('sdxl_image2image.png')原图像生成图像image_to_image_text_guided_generation-kandinsky2_2importpaddlefromppdiffusersimportKandinskyV22Img2ImgPipeline,KandinskyV22PriorPipelinefromppdiffusers.utilsimportload_imagepipe_prior=KandinskyV22PriorPipeline.from_pretrained("kandinsky-community/kandinsky-2-2-prior",paddle_dtype=paddle.float16)prompt="A red cartoon frog, 4k"image_emb,zero_image_emb=pipe_prior(prompt,return_dict=False)pipe=KandinskyV22Img2ImgPipeline.from_pretrained("kandinsky-community/kandinsky-2-2-decoder",paddle_dtype=paddle.float16)init_image=load_image("https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main""/kandinsky/frog.png")image=pipe(image=init_image,image_embeds=image_emb,negative_image_embeds=zero_image_emb,height=768,width=768,num_inference_steps=100,strength=0.2,).imagesimage[0].save("image_to_image_text_guided_generation-kandinsky2_2-result-red_frog.png")原图像生成图像文本图像双引导图像生成（Dual Text and Image Guided Generation）dual_text_and_image_guided_generation-versatile_diffusionfromppdiffusersimportVersatileDiffusionDualGuidedPipelinefromppdiffusers.utilsimportload_imageurl="https://paddlenlp.bj.bcebos.com/models/community/CompVis/data/benz.jpg"image=load_image(url)text="a red car in the sun"pipe=VersatileDiffusionDualGuidedPipeline.from_pretrained("shi-labs/versatile-diffusion")pipe.remove_unused_weights()text_to_image_strength=0.75image=pipe(prompt=text,image=image,text_to_image_strength=text_to_image_strength).images[0]image.save("versatile-diffusion-red_car.png")原图像生成图像文本视频多模文本条件的视频生成（Text-to-Video Generation）text_to_video_generation-lvdmimportpaddlefromppdiffusersimportLVDMTextToVideoPipeline# 加载模型和schedulerpipe=LVDMTextToVideoPipeline.from_pretrained("westfish/lvdm_text2video_orig_webvid_2m")# 执行pipeline进行推理seed=2013generator=paddle.Generator().manual_seed(seed)samples=pipe(prompt="cutting in kitchen",num_frames=16,height=256,width=256,num_inference_steps=50,generator=generator,guidance_scale=15,eta=1,save_dir=".",save_name="text_to_video_generation-lvdm-result-ddim_lvdm_text_to_video_ucf",encoder_type="2d",scale_factor=0.18215,shift_factor=0,)text_to_video_generation-synthimportimageiofromppdiffusersimportDPMSolverMultistepScheduler,TextToVideoSDPipelinepipe=TextToVideoSDPipeline.from_pretrained("damo-vilab/text-to-video-ms-1.7b")pipe.scheduler=DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)prompt="An astronaut riding a horse."video_frames=pipe(prompt,num_inference_steps=25).framesimageio.mimsave("text_to_video_generation-synth-result-astronaut_riding_a_horse.mp4",video_frames,fps=8)text_to_video_generation-synth with zeroscope_v2_XLimportimageiofromppdiffusersimportDPMSolverMultistepScheduler,TextToVideoSDPipeline# from ppdiffusers.utils import export_to_videopipe=TextToVideoSDPipeline.from_pretrained("cerspense/zeroscope_v2_XL")pipe.scheduler=DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)prompt="An astronaut riding a horse."video_frames=pipe(prompt,num_inference_steps=50,height=320,width=576,num_frames=24).framesimageio.mimsave("text_to_video_generation-synth-result-astronaut_riding_a_horse.mp4",video_frames,fps=8)text_to_video_generation-zeroimportimageio# pip install imageio[ffmpeg]importpaddlefromppdiffusersimportTextToVideoZeroPipelinemodel_id="runwayml/stable-diffusion-v1-5"pipe=TextToVideoZeroPipeline.from_pretrained(model_id,paddle_dtype=paddle.float16)prompt="A panda is playing guitar on times square"result=pipe(prompt=prompt).imagesresult=[(r*255).astype("uint8")forrinresult]imageio.mimsave("text_to_video_generation-zero-result-panda.mp4",result,fps=4)文本音频多模文本条件的音频生成（Text-to-Audio Generation）text_to_audio_generation-audio_ldmimportpaddleimportscipyfromppdiffusersimportAudioLDMPipelinepipe=AudioLDMPipeline.from_pretrained("cvssp/audioldm",paddle_dtype=paddle.float16)prompt="Techno music with a strong, upbeat tempo and high melodic riffs"audio=pipe(prompt,num_inference_steps=10,audio_length_in_s=5.0).audios[0]output_path="text_to_audio_generation-audio_ldm-techno.wav"# save the audio sample as a .wav filescipy.io.wavfile.write(output_path,rate=16000,data=audio)图像无条件图像生成（Unconditional Image Generation）unconditional_image_generation-latent_diffusion_uncondfromppdiffusersimportLDMPipeline# 加载模型和schedulerpipe=LDMPipeline.from_pretrained("CompVis/ldm-celebahq-256")# 执行pipeline进行推理image=pipe(num_inference_steps=200).images[0]# 保存图片image.save("ldm_generated_image.png")超分（Super Superresolution）super_resolution-latent_diffusionimportpaddlefromppdiffusersimportLDMSuperResolutionPipelinefromppdiffusers.utilsimportload_image# 加载pipelinepipe=LDMSuperResolutionPipeline.from_pretrained("CompVis/ldm-super-resolution-4x-openimages")# 下载初始图片url="https://paddlenlp.bj.bcebos.com/models/community/CompVis/stable-diffusion-v1-4/overture-creations.png"init_image=load_image(url).resize((128,128))init_image.save("original-image.png")# 使用fp16加快生成速度withpaddle.amp.auto_cast(True):image=pipe(init_image,num_inference_steps=100,eta=1).images[0]image.save("super-resolution-image.png")原图像生成图像图像编辑（Image Inpainting）image_inpainting-repaintfromppdiffusersimportRePaintPipeline,RePaintSchedulerfromppdiffusers.utilsimportload_imageimg_url="https://paddlenlp.bj.bcebos.com/models/community/CompVis/data/celeba_hq_256.png"mask_url="https://paddlenlp.bj.bcebos.com/models/community/CompVis/data/mask_256.png"# Load the original image and the mask as PIL imagesoriginal_image=load_image(img_url).resize((256,256))mask_image=load_image(mask_url).resize((256,256))scheduler=RePaintScheduler.from_pretrained("google/ddpm-ema-celebahq-256",subfolder="scheduler")pipe=RePaintPipeline.from_pretrained("google/ddpm-ema-celebahq-256",scheduler=scheduler)output=pipe(original_image=original_image,mask_image=mask_image,num_inference_steps=250,eta=0.0,jump_length=10,jump_n_sample=10,)inpainted_image=output.images[0]inpainted_image.save("repaint-image.png")原图像mask图像生成图像图像变化（Image Variation）image_variation-versatile_diffusionfromppdiffusersimportVersatileDiffusionImageVariationPipelinefromppdiffusers.utilsimportload_imageurl="https://paddlenlp.bj.bcebos.com/models/community/CompVis/data/benz.jpg"image=load_image(url)pipe=VersatileDiffusionImageVariationPipeline.from_pretrained("shi-labs/versatile-diffusion")image=pipe(image).images[0]image.save("versatile-diffusion-car_variation.png")原图像生成图像音频无条件音频生成（Unconditional Audio Generation）unconditional_audio_generation-audio_diffusionfromscipy.io.wavfileimportwritefromppdiffusersimportAudioDiffusionPipelineimportpaddle# 加载模型和schedulerpipe=AudioDiffusionPipeline.from_pretrained("teticio/audio-diffusion-ddim-256")pipe.set_progress_bar_config(disable=None)generator=paddle.Generator().manual_seed(42)output=pipe(generator=generator)audio=output.audios[0]image=output.images[0]# 保存音频到本地fori,audioinenumerate(audio):write(f"audio_diffusion_test{i}.wav",pipe.mel.sample_rate,audio.transpose())# 保存图片image.save("audio_diffusion_test.png")unconditional_audio_generation-spectrogram_diffusionimportpaddleimportscipyfromppdiffusersimportMidiProcessor,SpectrogramDiffusionPipelinefromppdiffusers.utils.download_utilsimportppdiffusers_url_download# Download MIDI from: wget https://paddlenlp.bj.bcebos.com/models/community/junnyu/develop/beethoven_hammerklavier_2.midmid_file_path=ppdiffusers_url_download("https://paddlenlp.bj.bcebos.com/models/community/junnyu/develop/beethoven_hammerklavier_2.mid",cache_dir=".")pipe=SpectrogramDiffusionPipeline.from_pretrained("google/music-spectrogram-diffusion",paddle_dtype=paddle.float16)processor=MidiProcessor()output=pipe(processor(mid_file_path))audio=output.audios[0]output_path="unconditional_audio_generation-spectrogram_diffusion-result-beethoven_hammerklavier_2.wav"# save the audio sample as a .wav filescipy.io.wavfile.write(output_path,rate=16000,data=audio)LicensePPDiffusers 遵循Apache-2.0开源协议。Stable Diffusion 遵循The CreativeML OpenRAIL M 开源协议。The CreativeML OpenRAIL M is anOpen RAIL M license, adapted from the work thatBigScienceandthe RAIL Initiativeare jointly carrying in the area of responsible AI licensing. See alsothe article about the BLOOM Open RAIL licenseon which this license is based.Acknowledge我们借鉴了🤗 Hugging Face的Diffusers关于预训练扩散模型使用的优秀设计，在此对Hugging Face作者及其开源社区表示感谢。Citation@misc{ppdiffusers,author={PaddlePaddle Authors},title={PPDiffusers: State-of-the-art diffusion model toolkit based on PaddlePaddle},year={2022},publisher={GitHub},journal={GitHub repository},howpublished={\url{https://github.com/PaddlePaddle/PaddleMIX/tree/develop/ppdiffusers}}}