Add Diffusers weights
This PR uploads the diffusers controlnet model for this repo. This allows users to use it within the diffusers ecosystem: https://github.com/huggingface/diffusers , and later also use the inference API here.
4 checkpoints are uploaded
- fp32 in torch.bin
- fp32 in safetensors
- fp16 in torch.bin
- fp16 in safetensors
Code to check the model:
import torch
import os
from diffusers.utils import load_image
from diffusers import (
ControlNetModel,
StableDiffusionControlNetPipeline,
)
image = load_image("https://huggingface.co/CrucibleAI/ControlNetMediaPipeFace/resolve/main/samples_laion_face_dataset/family_annotation.png")
controlnet = ControlNetModel.from_pretrained("CrucibleAI/ControlNetMediaPipeFace", torch_dtype=torch.float16, variant="fp16", revision="refs/pr/6")
pipe = StableDiffusionControlNetPipeline.from_pretrained(
"stabilityai/stable-diffusion-2-1-base", controlnet=controlnet, torch_dtype=torch.float16
)
pipe.enable_model_cpu_offload()
out_image = pipe("A happy family in a dentist advertisement.", image=image).images[0]
Running the above code gives
@chris-crucible I think this PR is good to go. I also added a short code snippet showing how the model can be used :-)
We're currently working on enabling the inference API for controlnet so adding the diffusers checkpoint should also help you run the inference API later
Excellent. It remains to make an ONNX model :)
LGTM. Outstanding work.
I have a deep-seated desire to move everything to the same directory level so the models are all in "/models" or "/", rather than a mix. I think you have a better understanding of HF best practices so I'll do a follow-up PR to style-match you.
That's interesting feedback @JosephCatrambone ! Sadly diffusers more or less requires to have nested folders now as it's the core of how pipeline components are handled.
🤷 No worries. I am not married to the idea of /models/*.pt.
In fact, I think it looks nicer without the extra folder. Thanks again for this PR.