Repository: oobabooga/text-generation-webui

Issue:
With multiple LoRAs loaded, only the first one in the list affects the generated text output
### Describe the bug

We're experimenting with training "stacked" LoRAs, as in LoRAs trained on top of other LoRAs. According to the warning message triggered by this, such an arrangement is supposed to work, but "may have unexpected effects". Training succeeds without any issue, but when I load the finished LoRAs, only the first one seems to have any effect.

So far, we have encountered and successfully worked around an issue where the 2nd LoRA's weights would never get loaded due to a PEFT bug/weirdness (the 2nd LoRA's weights get serialized with module names starting with "base_model.model.base_model.model.model", but on deserialization, PEFT expects "base_model.model.model"). Patching this made the 2nd LoRA load and affect the generated output successfully when it's the only LoRA being loaded. However, when we load it on top of the 1st LoRA, it still has no effect. With the loading order reversed compared to the training order, only the 2nd LoRA has any effect on the output - as if the 2nd and subsequent LoRAs were basically ignored.

To test this, we tried loading two known good, independently trained LoRAs (tloan/alpaca-lora-7b and 22h/cabrita-lora-v0-1) on a common base model (yahma/llama-7b-hf). Each LoRA works fine when loaded on its own, and is seen to affect the output.

When the "LoRA(s)" box is set to list alpaca first, then cabrita, both are shown as successfully loaded, but the generated text is identical to what I get with only alpaca loaded. If I put cabrita first, then alpaca, the output is indistinguishable from only cabrita being selected. This seems to confirm that only the first LoRA listed in the box is properly loaded into the model, the rest are silently ignored.

BnB quantization doesn't appear to affect this behavior - it's the same in 4-bit, 8-bit and native mode.

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Reproduction

Download yahma/llama-7b-hf, tloen/alpaca-lora-7b and 22h/cabrita-lora-v0-1 from Hugging Face.

Start the UI in chat mode.
On the Parameters tab, set the temperature to 0.3.
On the Chat settings tab, select the "Alpaca" instruction template.
On the Model tab, load the yahma/llama-7b-hf base model.

Without loading any LoRA, switch to the Text generation tab, and run the following 3 queries (clear history after each one):
- "Tell me about alpacas."
- "Invente uma desculpa criativa pra dizer que não preciso ir à festa."
- "How many helicopters can a human eat in one sitting?"

Note the responses. (The Portuguese query will prompt a response in English, while the helicopter one will be nonsense.)

On the Model tab, reload the model, then load the alpaca LoRA. Run the above 3 questions again. Note how the answers change. (The Portuguese prompt is now answered in Portuguese, but not very creatively, and the helicopter answer will be correct, i.e. none.)

Unload the alpaca LoRA, reload the model, and load the cabrita LoRA. Run the 3 prompts, and note the new answers. (Portuguese answer is different, more creative, while the helicopter one is back to nonsense.)

Unload the LoRA, reload the model. In the LoRA(s) box, select cabrita first, then alpaca. Run the 3 prompts. (Answers are as if only cabrita were loaded!)

Unload the LoRAs, reload the model. In the LoRA(s) box, select alpaca first, then cabrita. Run the 3 prompts. (Answers indistinguishable from when only alpaca was loaded!)

### Screenshot

_No response_

### Logs

```shell
no exception generated
```


### System Info

```shell
Ryzen 7950X with a Zotac GeForce RTX 4080 GPU, running Ubuntu 22.04 LTS with the open-source NVIDIA kernel module (with nvidia.NVreg_OpenRmEnableUnsupportedGpus=1 specified on the kernel command line to skip the "datacenter GPUs only" check). The issue is reproducible in both CUDA and CPU mode.
```



Discussion:
It's not a bug, it's a feature. 
There is no such thing as "stacked LoRA" You have it confused with Stable Diffusion (that does dynamically merge multiple LoRAs on forward during interference). But PEFT doesn't. PEFT uses one LoRA at a time.

If you add multiple LoRA's to PEFT model you can: 
1. quickly switch between them (regardless of Rank)
2. merge them into another LoRA (must have the same Rank)

My playground extension does all of those things and more.
https://github.com/FartyPants/Playground

Training LoRA on top of PEFT model+LoRA - absolutely NO.
You need to merge model with LoRA if you want to train on top of that for some reason (but it may not be what you imagine it to be). 

Resolving patch:
diff --git a/modules/LoRA.py b/modules/LoRA.py
--- a/modules/LoRA.py
+++ b/modules/LoRA.py
@@ -8,6 +8,14 @@
 from modules.models import reload_model
 
 
+def merge_loras():
+    if len(list({shared.model.peft_config[adapter].r for adapter in shared.model.peft_config.keys()})) > 1:
+        logger.warning("The loaded LoRAs cannot be merged, as they have dissimilar ranks. Only the first one will be active.")
+        return
+
+    shared.model.add_weighted_adapter(shared.lora_names, [1] * len(shared.lora_names), "__merged")
+    shared.model.set_adapter("__merged")
+
 def add_lora_to_model(lora_names):
     if 'GPTQForCausalLM' in shared.model.__class__.__name__ or shared.args.loader == 'AutoGPTQ':
         add_lora_autogptq(lora_names)
@@ -136,11 +144,14 @@ def add_lora_transformers(lora_names):
         return
 
     # Add a LoRA when another LoRA is already present
-    if len(removed_set) == 0 and len(prior_set) > 0:
+    if len(removed_set) == 0 and len(prior_set) > 0 and "__merged" not in shared.model.peft_config.keys():
         logger.info(f"Adding the LoRA(s) named {added_set} to the model...")
         for lora in added_set:
             shared.model.load_adapter(get_lora_path(lora), lora)
 
+        if len(lora_names) > 1:
+            merge_loras()
+
         return
 
     # If any LoRA needs to be removed, start over
@@ -165,6 +176,9 @@ def add_lora_transformers(lora_names):
         for lora in lora_names[1:]:
             shared.model.load_adapter(get_lora_path(lora), lora)
 
+        if len(lora_names) > 1:
+            merge_loras()
+
         shared.lora_names = lora_names
 
         if not shared.args.load_in_8bit and not shared.args.cpu:


Test patch: