Repository: nod-ai/SHARK

Issue:
Build 557 custom inpainting models
Through a lot of testing today of inpainting on a 7900xtx with build 557

I found that custom inpainting models currently only work when compiling from huggingface ID

Example model: ghunkins/stable-diffusion-liberty-inpainting

1. Put ghunkins/stable-diffusion-liberty-inpainting into the HuggingFace Model ID field
2. use a 512x512 input image and generate at default settings 512x512
3. profit

However if downloading the same inpainting model or any other custom model it doesn't work currently.

1. Download https://huggingface.co/ghunkins/stable-diffusion-liberty-inpainting/blob/main/liberty_main-inpainting.safetensors
2. Place it into ./models directory
4. Launch shark and select the model from drop down
5. use a 512x512 img and try generating at default settings 512x512
6. produces errors [safetensorsmodel error.txt](https://github.com/nod-ai/SHARK/files/10819746/safetensorsmodel.error.txt)

Now I have successfully got custom inpainting .safetensors models I've made in automatic1111 as well as downloaded models like the liberty-inpainting model and others working but only by using Onnx to extract the diffuser files, then copying those diffuser files into the ./models/(model_name)/ folder and having shark compile the model with the .safetensors file  and the diffusers onnx extracted. but that is obviously a bit too complicated.

I've also found that input image must match dimensions of the selection in the GUI. For example 512x512 default resolutions must also have input image of 512x512 as well. This is probably a separate "issue" but wanted to put this here to avoid any additional confusion/errors when testing.


Discussion:
From the error message looks like the base model it got is `stabilityai/stable-diffusion-2-1` instead of some inpainting model, so it had channel mismatch error. @jinchen62 may look at the details.

Resolving patch:
diff --git a/apps/stable_diffusion/scripts/img2img.py b/apps/stable_diffusion/scripts/img2img.py
--- a/apps/stable_diffusion/scripts/img2img.py
+++ b/apps/stable_diffusion/scripts/img2img.py
@@ -119,7 +119,7 @@ def img2img_inf(
         model_id = (
             args.hf_model_id
             if args.hf_model_id
-            else "runwayml/stable-diffusion-inpainting"
+            else "stabilityai/stable-diffusion-2-1-base"
         )
         schedulers = get_schedulers(model_id)
         scheduler_obj = schedulers[scheduler]
diff --git a/apps/stable_diffusion/src/utils/utils.py b/apps/stable_diffusion/src/utils/utils.py
--- a/apps/stable_diffusion/src/utils/utils.py
+++ b/apps/stable_diffusion/src/utils/utils.py
@@ -300,8 +300,6 @@ def set_init_device_flags():
         "stabilityai/stable-diffusion-2-1",
         "stabilityai/stable-diffusion-2-1-base",
         "CompVis/stable-diffusion-v1-4",
-        "runwayml/stable-diffusion-inpainting",
-        "stabilityai/stable-diffusion-2-inpainting",
     ]:
         args.import_mlir = True
 
@@ -439,10 +437,12 @@ def preprocessCKPT(custom_weights):
     print(
         "Loading diffusers' pipeline from original stable diffusion checkpoint"
     )
+    num_in_channels = 9 if "inpainting" in custom_weights else 4
     pipe = load_pipeline_from_original_stable_diffusion_ckpt(
         checkpoint_path=custom_weights,
         extract_ema=extract_ema,
         from_safetensors=from_safetensors,
+        num_in_channels=num_in_channels,
     )
     pipe.save_pretrained(path_to_diffusers)
     print("Loading complete")


Test patch: