Inpainting model
Are there plans to add an inpainting version of the model? Thanks.
Currently no plan. Could try controlnet based inpainting to see if it works well with lightning.
Thanks for your reply. There are many controlnet models for SDXL (https://huggingface.co/lllyasviel/sd_control_collection/tree/main) but unfortunately, there doesn't seem to be a controlnet for inpainting. There is an inpainting controlnet for SD1.5 but that doesn't work with SDXL or SDXL-lightning.
I looked into using checkpoint merging to create an inpainting version of SDXL-lightning, as described here: https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/How-to-make-your-own-Inpainting-model
I used the parameters below:
A = sd_xl_base_1.0_inpainting_0.1
B = sdxl_lightning_2step
C = sd_xl_base_1.0
M = 1
Unfortunately, the resulting SDXL-lightning inpainting model seems to require a large number of steps (like the original inpainting model) to get reasonable results, which defeats the purpose of using the lightning model.
If you have any other ideas that I could try, I would greatly appreciate it. Thank you.
Hey netgeek, regarding to sdxl inpainting, I have heard some good feedbacks from Fooocus' inpainting module, maybe you could give it a try.
Fortunately there is a comfyui port of the Fooocus inpainting capability, you could check it out at https://github.com/Acly/comfyui-inpaint-nodes/tree/main?tab=readme-ov-file#fooocus-inpaint
Thanks for the tip, seravee008. I'll take a look at it.