Repository: huggingface/diffusers
Issue category: Bug Report

ControlNet broken from_single_file
### Describe the bug

controlnet loader from_single_file was originally added via #4084
and method `ControlNet.from_single_file()` works for non-converted controlnets.

but for controlnets in safetensors format that contain already converted state_dict, it errors out.

its not reasonable to expect from user to know what is the internal dict structure of the controlnet safetensors file  
before he can use it.

even worse, some of the newer controlnets are distributed as single-file-only and are already in diffusers format  
which makes them impossible to load in difufsers.
for example: <https://huggingface.co/Laxhar/noob_openpose/tree/main>

this issue was already mentioned several times, each time closed as "works as designed"  
when in reality its just a failure that should be addressed as an issue.  
see #8474 #9208 #8614 as examples of previous issues

### Reproduction

scenario-1: works with non-converted controlnet
```python
import torch
from diffusers import ControlNetModel
from huggingface_hub import hf_hub_download
local_path = hf_hub_download(repo_id='Aptronym/SDNext', filename='ControlNet11/controlnet11Models_canny.safetensors')
cn = ControlNetModel.from_single_file(local_path, torch_dtype=torch.float16)
print(cn.__class__)
```

scenario-1: fails for majority of controlnets available on huggingface
```python
import torch
from diffusers import ControlNetModel
from huggingface_hub import hf_hub_download
local_path = hf_hub_download(repo_id='lllyasviel/sd_control_collection', filename='diffusers_xl_canny_small.safetensors')
cn = ControlNetModel.from_single_file(local_path, torch_dtype=torch.float16)
print(cn.__class__)
```
initial failure is nonsense
> OSError: stable-diffusion-v1-5/stable-diffusion-v1-5 does not appear to have a file named config.json.

whats making this worse is that SD15 and SDXL share the same `ControlNet` class which causes some
confusion on the base repo where to lookup config.
e.g,, here we're loading SDXL controlnet and error referrs to SD15 repo.

anyhow, trying to force correct config:
```py
cn = ControlNetModel.from_single_file(local_path, torch_dtype=torch.float16, config='diffusers/controlnet-canny-sdxl-1.0-small')
```

results in even worse nonsense failure during loading of state_dict:
> TypeError: is_floating_point(): argument 'input' (position 1) must be Tensor, not NoneType

### System Info

diffusers=0.32.0.dev0
python==3.12.3
torch==2.5.1+cu124

### Who can help?

@yiyixuxu @sayakpaul @DN6 @asomoza


Functions to edit:
src/diffusers/loaders/single_file_utils.py:infer_diffusers_model_type
src/diffusers/loaders/single_file_utils.py:convert_controlnet_checkpoint

Patch:
diff --git a/src/diffusers/loaders/single_file_utils.py b/src/diffusers/loaders/single_file_utils.py
index d1bad8b5a7cd..9a460cb5d1ef 100644
--- a/src/diffusers/loaders/single_file_utils.py
+++ b/src/diffusers/loaders/single_file_utils.py
@@ -62,7 +62,14 @@
     "xl_base": "conditioner.embedders.1.model.transformer.resblocks.9.mlp.c_proj.bias",
     "xl_refiner": "conditioner.embedders.0.model.transformer.resblocks.9.mlp.c_proj.bias",
     "upscale": "model.diffusion_model.input_blocks.10.0.skip_connection.bias",
-    "controlnet": "control_model.time_embed.0.weight",
+    "controlnet": [
+        "control_model.time_embed.0.weight",
+        "controlnet_cond_embedding.conv_in.weight",
+    ],
+    # TODO: find non-Diffusers keys for controlnet_xl
+    "controlnet_xl": "add_embedding.linear_1.weight",
+    "controlnet_xl_large": "down_blocks.1.attentions.0.transformer_blocks.0.attn1.to_k.weight",
+    "controlnet_xl_mid": "down_blocks.1.attentions.0.norm.weight",
     "playground-v2-5": "edm_mean",
     "inpainting": "model.diffusion_model.input_blocks.0.0.weight",
     "clip": "cond_stage_model.transformer.text_model.embeddings.position_embedding.weight",
@@ -96,6 +103,9 @@
     "inpainting": {"pretrained_model_name_or_path": "stable-diffusion-v1-5/stable-diffusion-inpainting"},
     "inpainting_v2": {"pretrained_model_name_or_path": "stabilityai/stable-diffusion-2-inpainting"},
     "controlnet": {"pretrained_model_name_or_path": "lllyasviel/control_v11p_sd15_canny"},
+    "controlnet_xl_large": {"pretrained_model_name_or_path": "diffusers/controlnet-canny-sdxl-1.0"},
+    "controlnet_xl_mid": {"pretrained_model_name_or_path": "diffusers/controlnet-canny-sdxl-1.0-mid"},
+    "controlnet_xl_small": {"pretrained_model_name_or_path": "diffusers/controlnet-canny-sdxl-1.0-small"},
     "v2": {"pretrained_model_name_or_path": "stabilityai/stable-diffusion-2-1"},
     "v1": {"pretrained_model_name_or_path": "stable-diffusion-v1-5/stable-diffusion-v1-5"},
     "stable_cascade_stage_b": {"pretrained_model_name_or_path": "stabilityai/stable-cascade", "subfolder": "decoder"},
@@ -481,8 +491,16 @@ def infer_diffusers_model_type(checkpoint):
     elif CHECKPOINT_KEY_NAMES["upscale"] in checkpoint:
         model_type = "upscale"
 
-    elif CHECKPOINT_KEY_NAMES["controlnet"] in checkpoint:
-        model_type = "controlnet"
+    elif any(key in checkpoint for key in CHECKPOINT_KEY_NAMES["controlnet"]):
+        if CHECKPOINT_KEY_NAMES["controlnet_xl"] in checkpoint:
+            if CHECKPOINT_KEY_NAMES["controlnet_xl_large"] in checkpoint:
+                model_type = "controlnet_xl_large"
+            elif CHECKPOINT_KEY_NAMES["controlnet_xl_mid"] in checkpoint:
+                model_type = "controlnet_xl_mid"
+            else:
+                model_type = "controlnet_xl_small"
+        else:
+            model_type = "controlnet"
 
     elif (
         CHECKPOINT_KEY_NAMES["stable_cascade_stage_c"] in checkpoint
@@ -1072,6 +1090,9 @@ def convert_controlnet_checkpoint(
     config,
     **kwargs,
 ):
+    # Return checkpoint if it's already been converted
+    if "time_embedding.linear_1.weight" in checkpoint:
+        return checkpoint
     # Some controlnet ckpt files are distributed independently from the rest of the
     # model components i.e. https://huggingface.co/thibaud/controlnet-sd21/
     if "time_embed.0.weight" in checkpoint: