Repository: huggingface/diffusers

Issue:
Unable to load Automatic1111 LoRa due rank being higher than "min(in_features, out_features)}"
### Describe the bug

Hi! I'm unable to load a LoRa as I get the following error.

`ValueError: LoRA rank 384 must be less or equal than 320`

The LoRa works perfectly if the ValueError check is removed from the code. Is it possible to override this error? I don't understand why it must be less or equal. Thanks!

### Reproduction

Run the interference script with [this](https://civitai.com/models/7227/urushi) LoRa
Get error about the LoRa rank

### Logs

```
  File "/home/wizard/pipelines/utils.py", line 465, in load_lora_weights_diffusers
    loader.load_lora_into_unet(state_dict, network_alphas=network_alphas, unet=unet)
  File "/home/wizard/lib/python3.9/site-packages/diffusers/loaders.py", line 1233, in load_lora_into_unet
    unet.load_attn_procs(state_dict, network_alphas=network_alphas)
  File "/home/wizardlib/python3.9/site-packages/diffusers/loaders.py", line 376, in load_attn_procs
    lora = LoRAConv2dLayer(
  File "/home/wizard/lib/python3.9/site-packages/diffusers/models/lora.py", line 58, in __init__
    raise ValueError(f"LoRA rank {rank} must be less or equal than {min(in_features, out_features)}")
```


### System Info

- `diffusers` version: 0.19.3
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.9.16
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Huggingface_hub version: 0.16.4
- Transformers version: 4.29.2
- Accelerate version: 0.16.0
- xFormers version: 0.0.20
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@williamberman @patrickvonplaten @saya


Discussion:
Cc: @isidentical what do you think?
> Run the interference script with [this](https://civitai.com/models/7227/urushi) LoRa

What is the base model you are testing with @sinanhlvc? Is it regular SD 1.5 or something else? 
> > Run the interference script with [this](https://civitai.com/models/7227/urushi) LoRa
> 
> What is the base model you are testing with @sinanhlvc? Is it regular SD 1.5 or something else?

Hi! Sorry I should've mentioned it. I have tested on base SD 1.5 and fine-tuned versions like Dreamshaper, deliberate, etc. and got the same error with this LoRa
Possible to post a fully reproducible snippet? Preferably a Colab Notebook would be really helpful. 

@isidentical WDYT?
> Possible to post a fully reproducible snippet? Preferably a Colab Notebook would be really helpful.
> 
> @isidentical WDYT?

[Here's](https://colab.research.google.com/drive/1JiaDKnUoV-Xhyxa2MGfVnGdBDrI132s2#scrollTo=AMLZh3hbrS5X&line=1&uniqifier=1) the notebook.
Looking at both A1111 and Kohya-SS's implementation, I think this is an allowed operation (where the LoRA rank outmatches the existing dimensions) (for both conv and linear):
https://github.com/huggingface/diffusers/blob/aaef41b5fe06535fe6dd0caa723386c0e7494cec/src/diffusers/models/lora.py#L57-L58

E.g. this is A1111s:

https://github.com/AUTOMATIC1111/stable-diffusion-webui/blob/68f336bd994bed5442ad95bad6b6ad5564a5409a/extensions-builtin/Lora/network_lora.py#L35-L55

As long as the different shapes don't create a problem (not super sure whether torch or diffusers automatically reshapes somewhere else), we might be able to drop this check @sayakpaul.

Yes this check shouldn't be necessary as the only requirement is the shapes hook together like 

in_dim -> rank -> rank -> out_dim
|------down----|   |--------up-----|

Though if rank is larger than both in dim and out dim technically the inner projection isn't a down projection anymore :P and you lose the nice properties of the auxiliary lora blocks being smaller than the original network.


Resolving patch:
diff --git a/src/diffusers/models/lora.py b/src/diffusers/models/lora.py
--- a/src/diffusers/models/lora.py
+++ b/src/diffusers/models/lora.py
@@ -22,9 +22,6 @@ class LoRALinearLayer(nn.Module):
     def __init__(self, in_features, out_features, rank=4, network_alpha=None, device=None, dtype=None):
         super().__init__()
 
-        if rank > min(in_features, out_features):
-            raise ValueError(f"LoRA rank {rank} must be less or equal than {min(in_features, out_features)}")
-
         self.down = nn.Linear(in_features, rank, bias=False, device=device, dtype=dtype)
         self.up = nn.Linear(rank, out_features, bias=False, device=device, dtype=dtype)
         # This value has the same meaning as the `--network_alpha` option in the kohya-ss trainer script.
@@ -54,9 +51,6 @@ def __init__(
     ):
         super().__init__()
 
-        if rank > min(in_features, out_features):
-            raise ValueError(f"LoRA rank {rank} must be less or equal than {min(in_features, out_features)}")
-
         self.down = nn.Conv2d(in_features, rank, kernel_size=kernel_size, stride=stride, padding=padding, bias=False)
         # according to the official kohya_ss trainer kernel_size are always fixed for the up layer
         # # see: https://github.com/bmaltais/kohya_ss/blob/2accb1305979ba62f5077a23aabac23b4c37e935/networks/lora_diffusers.py#L129


Test patch:
diff --git a/tests/models/test_lora_layers.py b/tests/models/test_lora_layers.py
--- a/tests/models/test_lora_layers.py
+++ b/tests/models/test_lora_layers.py
@@ -742,6 +742,25 @@ def test_a1111(self):
 
         self.assertTrue(np.allclose(images, expected, atol=1e-4))
 
+    def test_kohya_sd_v15_with_higher_dimensions(self):
+        generator = torch.Generator().manual_seed(0)
+
+        pipe = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", safety_checker=None).to(
+            torch_device
+        )
+        lora_model_id = "hf-internal-testing/urushisato-lora"
+        lora_filename = "urushisato_v15.safetensors"
+        pipe.load_lora_weights(lora_model_id, weight_name=lora_filename)
+
+        images = pipe(
+            "masterpiece, best quality, mountain", output_type="np", generator=generator, num_inference_steps=2
+        ).images
+
+        images = images[0, -3:, -3:, -1].flatten()
+        expected = np.array([0.7165, 0.6616, 0.5833, 0.7504, 0.6718, 0.587, 0.6871, 0.6361, 0.5694])
+
+        self.assertTrue(np.allclose(images, expected, atol=1e-4))
+
     def test_vanilla_funetuning(self):
         generator = torch.Generator().manual_seed(0)