Repository: huggingface/diffusers
Issue category: Bug Report

Issue:
ControlNet broken from_single_file
### Describe the bug

controlnet loader from_single_file was originally added via #4084
and method `ControlNet.from_single_file()` works for non-converted controlnets.

but for controlnets in safetensors format that contain already converted state_dict, it errors out.

its not reasonable to expect from user to know what is the internal dict structure of the controlnet safetensors file  
before he can use it.

even worse, some of the newer controlnets are distributed as single-file-only and are already in diffusers format  
which makes them impossible to load in difufsers.
for example: <https://huggingface.co/Laxhar/noob_openpose/tree/main>

this issue was already mentioned several times, each time closed as "works as designed"  
when in reality its just a failure that should be addressed as an issue.  
see #8474 #9208 #8614 as examples of previous issues

### Reproduction

scenario-1: works with non-converted controlnet
```python
import torch
from diffusers import ControlNetModel
from huggingface_hub import hf_hub_download
local_path = hf_hub_download(repo_id='Aptronym/SDNext', filename='ControlNet11/controlnet11Models_canny.safetensors')
cn = ControlNetModel.from_single_file(local_path, torch_dtype=torch.float16)
print(cn.__class__)
```

scenario-1: fails for majority of controlnets available on huggingface
```python
import torch
from diffusers import ControlNetModel
from huggingface_hub import hf_hub_download
local_path = hf_hub_download(repo_id='lllyasviel/sd_control_collection', filename='diffusers_xl_canny_small.safetensors')
cn = ControlNetModel.from_single_file(local_path, torch_dtype=torch.float16)
print(cn.__class__)
```
initial failure is nonsense
> OSError: stable-diffusion-v1-5/stable-diffusion-v1-5 does not appear to have a file named config.json.

whats making this worse is that SD15 and SDXL share the same `ControlNet` class which causes some
confusion on the base repo where to lookup config.
e.g,, here we're loading SDXL controlnet and error referrs to SD15 repo.

anyhow, trying to force correct config:
```py
cn = ControlNetModel.from_single_file(local_path, torch_dtype=torch.float16, config='diffusers/controlnet-canny-sdxl-1.0-small')
```

results in even worse nonsense failure during loading of state_dict:
> TypeError: is_floating_point(): argument 'input' (position 1) must be Tensor, not NoneType

### System Info

diffusers=0.32.0.dev0
python==3.12.3
torch==2.5.1+cu124

### Who can help?

@yiyixuxu @sayakpaul @DN6 @asomoza


Hints:
> even worse, some of the newer controlnets are distributed as single-file-only and are already in diffusers format
which makes them impossible to load in difufsers.
for example: https://huggingface.co/Laxhar/noob_openpose/tree/main

Isn't this an actual error as it is partially in the diffusers format by not including the `config.json`? 
@sayakpaul cannot blame users for not knowing internals of `diffusers`.
99% of users will attempt to download that safetensors file and load it - and it will fail.
sure, author of that specific repo should add `config.json` (and yes, request to author is already created),  
but are we going to re-educate every user "don't download controlnet safetensors"?
(and there is no way to differentiate from user perspective which ones can and which cannot be downloaded)
In that case what looks like a good solution to you? I will let @DN6 to of course comment here but IMO the following could be good candidates: 

* Detect if this is a `diffusers` checkpoint and if the supplied path doesn't have a `config.json`. If detected, yield a nice error message. 
* Detect if this is a `diffusers` checkpoint and try to fetch a `config.json` but I am not sure how well this would play out as it involves quite a bit of guess work from what I can see. 
> Detect if this is a diffusers checkpoint and try to fetch a config.json but I am not sure how well this would play out as it involves quite a bit of guess work from what I can see.

which guesswork? is it sd15 or sdxl should be easy to infer from state_dict itself.
that's how it works when loading main dffusion model using from_single_file. if i were a user, i would have same expectation here.
Hmm could be. I thought too fast. So, fetch the config, load the state dict -- looks like the workflow here. 

```py
with init_empty_weights():
    config = ControlNetModel.load_config("...")
    controlnet_model = ControlNetModel.from_config(config)
    controlnet_model.load_state_dict(diffusers_state_dict)
```
> Hmm could be. I thought too fast. So, fetch the config, load the state dict -- looks like the workflow here.

btw, as a starting point, we can pass `config=...` arg to `from_single_file` and add autodetect later.
but need to make `from_single_file` actually work :)

In the first case the model config is being detected as `v1` because the checked keys for `controlnet` are for non-Diffusers type checkpoints.

https://github.com/huggingface/diffusers/blob/f6f7afa1d7c6f45f8568c5603b1e6300d4583f04/src/diffusers/loaders/single_file_utils.py#L457

https://github.com/huggingface/diffusers/blob/f6f7afa1d7c6f45f8568c5603b1e6300d4583f04/src/diffusers/loaders/single_file_utils.py#L65

`infer_diffusers_model_type` can also be extended to detect e.g. `v1` vs `xl` ControlNet.

In the second case `convert_controlnet_checkpoint` expects non-Diffusers type, the state dict it returns appears to only contain `controlnet_mid_block.weight` and `controlnet_mid_block.bias` as `None` which in turn causes the error when the state dict is loaded to the model.

`convert_controlnet_checkpoint` can return the provided checkpoint when Diffusers keys are detected.

https://github.com/huggingface/diffusers/blob/f6f7afa1d7c6f45f8568c5603b1e6300d4583f04/src/diffusers/loaders/single_file_utils.py#L1169-L1170

https://github.com/huggingface/diffusers/blob/f6f7afa1d7c6f45f8568c5603b1e6300d4583f04/src/diffusers/loaders/single_file_utils.py#L1070

https://github.com/huggingface/diffusers/blob/f6f7afa1d7c6f45f8568c5603b1e6300d4583f04/src/diffusers/models/model_loading_utils.py#L170


Patch:
diff --git a/src/diffusers/loaders/single_file_utils.py b/src/diffusers/loaders/single_file_utils.py
index d1bad8b5a7cd..9a460cb5d1ef 100644
--- a/src/diffusers/loaders/single_file_utils.py
+++ b/src/diffusers/loaders/single_file_utils.py
@@ -62,7 +62,14 @@
     "xl_base": "conditioner.embedders.1.model.transformer.resblocks.9.mlp.c_proj.bias",
     "xl_refiner": "conditioner.embedders.0.model.transformer.resblocks.9.mlp.c_proj.bias",
     "upscale": "model.diffusion_model.input_blocks.10.0.skip_connection.bias",
-    "controlnet": "control_model.time_embed.0.weight",
+    "controlnet": [
+        "control_model.time_embed.0.weight",
+        "controlnet_cond_embedding.conv_in.weight",
+    ],
+    # TODO: find non-Diffusers keys for controlnet_xl
+    "controlnet_xl": "add_embedding.linear_1.weight",
+    "controlnet_xl_large": "down_blocks.1.attentions.0.transformer_blocks.0.attn1.to_k.weight",
+    "controlnet_xl_mid": "down_blocks.1.attentions.0.norm.weight",
     "playground-v2-5": "edm_mean",
     "inpainting": "model.diffusion_model.input_blocks.0.0.weight",
     "clip": "cond_stage_model.transformer.text_model.embeddings.position_embedding.weight",
@@ -96,6 +103,9 @@
     "inpainting": {"pretrained_model_name_or_path": "stable-diffusion-v1-5/stable-diffusion-inpainting"},
     "inpainting_v2": {"pretrained_model_name_or_path": "stabilityai/stable-diffusion-2-inpainting"},
     "controlnet": {"pretrained_model_name_or_path": "lllyasviel/control_v11p_sd15_canny"},
+    "controlnet_xl_large": {"pretrained_model_name_or_path": "diffusers/controlnet-canny-sdxl-1.0"},
+    "controlnet_xl_mid": {"pretrained_model_name_or_path": "diffusers/controlnet-canny-sdxl-1.0-mid"},
+    "controlnet_xl_small": {"pretrained_model_name_or_path": "diffusers/controlnet-canny-sdxl-1.0-small"},
     "v2": {"pretrained_model_name_or_path": "stabilityai/stable-diffusion-2-1"},
     "v1": {"pretrained_model_name_or_path": "stable-diffusion-v1-5/stable-diffusion-v1-5"},
     "stable_cascade_stage_b": {"pretrained_model_name_or_path": "stabilityai/stable-cascade", "subfolder": "decoder"},
@@ -481,8 +491,16 @@ def infer_diffusers_model_type(checkpoint):
     elif CHECKPOINT_KEY_NAMES["upscale"] in checkpoint:
         model_type = "upscale"
 
-    elif CHECKPOINT_KEY_NAMES["controlnet"] in checkpoint:
-        model_type = "controlnet"
+    elif any(key in checkpoint for key in CHECKPOINT_KEY_NAMES["controlnet"]):
+        if CHECKPOINT_KEY_NAMES["controlnet_xl"] in checkpoint:
+            if CHECKPOINT_KEY_NAMES["controlnet_xl_large"] in checkpoint:
+                model_type = "controlnet_xl_large"
+            elif CHECKPOINT_KEY_NAMES["controlnet_xl_mid"] in checkpoint:
+                model_type = "controlnet_xl_mid"
+            else:
+                model_type = "controlnet_xl_small"
+        else:
+            model_type = "controlnet"
 
     elif (
         CHECKPOINT_KEY_NAMES["stable_cascade_stage_c"] in checkpoint
@@ -1072,6 +1090,9 @@ def convert_controlnet_checkpoint(
     config,
     **kwargs,
 ):
+    # Return checkpoint if it's already been converted
+    if "time_embedding.linear_1.weight" in checkpoint:
+        return checkpoint
     # Some controlnet ckpt files are distributed independently from the rest of the
     # model components i.e. https://huggingface.co/thibaud/controlnet-sd21/
     if "time_embed.0.weight" in checkpoint:


Test patch: