Repository: huggingface/peft

Issue:
'set_adapter()' throws "ValueError: Adapter not found in odict_keys" after 'load_adapter()'
### System Info

peft-0.9.0

### Who can help?

@pacman100 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder
- [ ] My own task or dataset (give details below)

### Reproduction
Saved two finetuned models to `peft_model_id_1` and `peft_model_id_2` in this way:

```
model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased")

if training_args.do_train:
    """ Add PEFT """
    peft_config = LoraConfig(
        task_type=TaskType.SEQ_CLS, 
        r=1, 
        lora_alpha=1, 
        lora_dropout=0.1
    )
    model = get_peft_model(model, peft_config)

    trainer = Trainer(
            model=model,
            args=training_args,
            train_dataset=train_dataset if training_args.do_train else None,
            eval_dataset=eval_dataset if training_args.do_eval else None,
            compute_metrics=compute_metrics,
            tokenizer=tokenizer,
            data_collator=data_collator,
        )
    
    train_result = trainer.train()

    peft_model_id = training_args.output_dir
    trainer.model.save_pretrained(peft_model_id)
    tokenizer.save_pretrained(peft_model_id)
```

Load the saved two adapters:
```
##################  load ####################
base_model = AutoModelForSequenceClassification.from_pretrained(
        "bert-base-uncased"
    )

merged_model = PeftModel.from_pretrained(base_model, peft_model_id_1, adapter_name="id_1")
merged_model.load_adapter(peft_model_id_2, adapter_name="id_2")
```

When trying to set adapter:
```
merged_model.set_adapter("id_2")
```

Got error:
```
ValueError: Adapter id_2 not found in odict_keys(['id_1'])
```

### Expected behavior

- I have checked `merged_model.peft_config` and got:
```
{'id_1': LoraConfig(peft_type...ora=False), 'id_2': LoraConfig(peft_type...ora=False)}
```

and both two adapters occur in `merged_mode`l if ```print(merged_model)```.
- `merged_model.add_weighted_adapter` can also work. 
- Only `merged_model.set_adapter("id_2")` has problems: `ValueError: Adapter id_2 not found in odict_keys(['id_1'])`. It seems that only the first adapter can be seen in `odict_keys`


Discussion:
That's indeed very strange, it looks like the steps you take are indeed correct. Could you please paste the full error message? Is it possible to access your adapters publicly somewhere so that I can try to reproduce?
Sure, here is the full error message, just several lines:
```
Exception has occurred: ValueError
Adapter id_2 not found in odict_keys(['id_1'])
  File "xxxx/code/merge_weight_peft.py", line 169, in main
    merged_model.set_adapter('id_2')
  File "xxxx/code/merge_weight_peft.py", line 207, in <module>
    app.run(main)
ValueError: Adapter id_2 not found in odict_keys(['id_1'])
```
I have upload the adapter_model.bin of two models I want to merge to `YnezT/backdoor-sst2-bert-base-uncased` and `YnezT/clean-sst2-bert-base-uncased`, respectively. What I'm trying to do is to merge this two and get a merged_model, then set that to the main adapter:

```
base_model = AutoModelForSequenceClassification.from_pretrained(
        "bert-base-uncased"
    )

merged_model = PeftModel.from_pretrained(base_model, backdoor_model_adapter, adapter_name="backdoor_model")
merged_model.load_adapter(clean_model_adapter, adapter_name="clean_model")
if merge_type == "ties-merge":
      """ ties-merge"""
      adapters = ["clean_model", "backdoor_model"]
      weights = [1.0, 1.0]
      adapter_name = "merge_model"
      density = 0.2
      combination_type = "ties"
      merged_model.add_weighted_adapter(adapters, weights, adapter_name, combination_type=combination_type, density=density)
      merged_model.set_adapter(adapter_name)
```

Thanks a lot!
Thanks for the additional context. However, I could not reproduce the error. Below is a code snippet. Note that I used my own LoRA adapters, as yours are .bin files and not safetensors (just as a safety measure), but that shouldn't really change things.

```python
from transformers import AutoModelForSequenceClassification
from peft import get_peft_model, LoraConfig, PeftModel

# creating adapters
config0 = LoraConfig(init_lora_weights=False)
config1 = LoraConfig(init_lora_weights=False)
base_model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased")
peft_model = get_peft_model(base_model, config0)
peft_model.add_adapter("other", config1)
peft_model.save_pretrained("/tmp/peft/bert")

# same as your code
base_model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased")
merged_model = PeftModel.from_pretrained(base_model, "/tmp/peft/bert/", adapter_name="backdoor_model")
merged_model.load_adapter("/tmp/peft/bert/other", adapter_name="clean_model")
adapters = ["clean_model", "backdoor_model"]
weights = [1.0, 1.0]
adapter_name = "merge_model"
density = 0.2
combination_type = "ties"
merged_model.add_weighted_adapter(adapters, weights, adapter_name, combination_type=combination_type, density=density)
merged_model.set_adapter(adapter_name)  # works
print(merged_model.active_adapters)  # shows ['merge_model']
```

What's also strange is that your error message refers to "id_1" and "id_2" but your code uses different adapter names. Are you sure that there isn't something else going on?
Sorry, I just manually changed the name in the error msg for consistency with the initial query, but forgot to change the later code. Here id_1 stands for backdoor_model and id_2 for merge_model. 

Could I ask for your log and Pytorch version? When running my code, I saw this:
```
Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForSequenceClassification: ['cls.predictions.transform.dense.bias', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.bias', 'cls.seq_relationship.weight']
- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```
Not sure whether it is related.
I have torch v2.2.0, transformers v4.39.0 and latest PEFT.

My log is a bit different to yours:

> Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight']
> You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

This is expected, since we use `AutoModelForSequenceClassification`, so a classification layer is added, which is untrained. When you do your fine-tuning, it should be added to `modules_to_save` in your `LoraConfig`. Regardless of that, I don't see how the error you described could possibly occur.
Apologies for my late reply; I was preoccupied with something else last week. 

I notice the differences between your code and mine.
```
config0 = LoraConfig(init_lora_weights=False)
config1 = LoraConfig(init_lora_weights=False)
base_model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased")
peft_model = get_peft_model(base_model, config0)
peft_model.add_adapter("other", config1)
peft_model.save_pretrained("/tmp/peft/bert")
```
In your code, here, these two adapters are used for training the "same" model, i.e., the `.save_pretrained` will save these two adapters together. Is this required for merging?

What I did is somewhat similar to:
```
config0 = LoraConfig(init_lora_weights=False)
base_model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased")
peft_model = get_peft_model(base_model, config0)
peft_model.save_pretrained("/tmp/peft/bert/main")
```
and
```
config1 = LoraConfig(init_lora_weights=False)
base_model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased")
peft_model = get_peft_model(base_model, config1)
peft_model.save_pretrained("/tmp/peft/bert/other")
```
and then
```
base_model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased")
merged_model = PeftModel.from_pretrained(base_model, "/tmp/peft/bert/main", adapter_name="backdoor_model")
merged_model.load_adapter("/tmp/peft/bert/other", adapter_name="clean_model")
```
I guess this is where the problem I encountered arises?
I have tested it, and it appears not to be the reason. Both work. I might figure out the problem myself. Thanks!
> In your code, here, these two adapters are used for training the "same" model, i.e., the `.save_pretrained` will save these two adapters together. Is this required for merging?

> I have tested it, and it appears not to be the reason.

I just want to confirm that this is not necessary and should not be the reason for the problem you encounter.

> I might figure out the problem myself. Thanks!

Feel free to share new insights or questions that you may have in this issue.
Thank you for your patience. I have identified where the problem lies, but I do not know how to solve it: If we specify the `task_type` in `LoraConfig`, an error occurs. For example, the following code will throw out an error.
```
from transformers import AutoModelForSequenceClassification
from peft import get_peft_model, LoraConfig, PeftModel, TaskType

# creating adapters
config0 = LoraConfig(task_type=TaskType.SEQ_CLS, init_lora_weights=False)
config1 = LoraConfig(task_type=TaskType.SEQ_CLS, init_lora_weights=False)
base_model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased")
peft_model = get_peft_model(base_model, config0)
peft_model.add_adapter("other", config1)
peft_model.save_pretrained("tmp/peft/bert/")

# same as your code
base_model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased")
merged_model = PeftModel.from_pretrained(base_model, "tmp/peft/bert/", adapter_name="backdoor_model")
merged_model.load_adapter("tmp/peft/bert/other", adapter_name="clean_model")
adapters = ["clean_model", "backdoor_model"]
weights = [1.0, 1.0]
adapter_name = "merge_model"
density = 0.2
combination_type = "ties"
merged_model.add_weighted_adapter(adapters, weights, adapter_name, combination_type=combination_type, density=density)
merged_model.set_adapter(adapter_name) # ValueError: Adapter merge_model not found in odict_keys(['backdoor_model'])
print(merged_model.active_adapters)
```
However, if we remove the `task_type`, the evaluation metric mismatch occurs. My current solution is to disable the evaluation phase in `Trainer`, and evaluate at the end by myself, but I assume there should be more standard solutions.
```
Exception has occurred: KeyError
'eval_loss'
  File "xxxx", line 127, in <module>
    clean_trainer.train()
KeyError: 'eval_loss'
```
Thanks for digging further. I can reproduce the error and this is indeed caused by a bug in PEFT, which actually runs deeper than what is reported in this issue. I'll work on a bugfix and will get back to you once it's ready.

Resolving patch:
diff --git a/src/peft/peft_model.py b/src/peft/peft_model.py
--- a/src/peft/peft_model.py
+++ b/src/peft/peft_model.py
@@ -698,7 +698,7 @@ def set_additional_trainable_modules(self, peft_config, adapter_name):
                 self.modules_to_save = set(peft_config.modules_to_save)
             else:
                 self.modules_to_save.update(peft_config.modules_to_save)
-            _set_trainable(self, adapter_name)
+            _set_trainable(self, adapter_name)  # this may add a new ModulesToSaveWrapper
 
     @classmethod
     def _split_kwargs(cls, kwargs: dict[str, Any]):
@@ -714,7 +714,7 @@ def _split_kwargs(cls, kwargs: dict[str, Any]):
 
         return hf_hub_download_kwargs, other_kwargs
 
-    def _update_offload(self, offload_index: dict[dict[str:str]], adapters_weights: dict[str : torch.tensor]):
+    def _update_offload(self, offload_index: dict[str, dict[str, str]], adapters_weights: dict[str, torch.tensor]):
         """
         Update the offload_index and safetensors files for loading and mergine PeftModels with disk-offloaded modules.
 
@@ -1023,19 +1023,54 @@ class PeftModelForSequenceClassification(PeftModel):
 
     def __init__(self, model: torch.nn.Module, peft_config: PeftConfig, adapter_name: str = "default") -> None:
         super().__init__(model, peft_config, adapter_name)
+
+        classifier_module_names = ["classifier", "score"]
         if self.modules_to_save is None:
-            self.modules_to_save = {"classifier", "score"}
+            self.modules_to_save = set(classifier_module_names)
         else:
-            self.modules_to_save.update({"classifier", "score"})
+            self.modules_to_save.update(classifier_module_names)
+
+        if hasattr(peft_config, "modules_to_save"):
+            if peft_config.modules_to_save is None:
+                peft_config.modules_to_save = classifier_module_names[:]
+            else:
+                peft_config.modules_to_save.extend(classifier_module_names)
 
         for name, _ in self.base_model.named_children():
             if any(module_name in name for module_name in self.modules_to_save):
                 self.cls_layer_name = name
                 break
 
-        # to make sure classifier layer is trainable
+        # to make sure classifier layer is trainable; this may add a new ModulesToSaveWrapper
         _set_trainable(self, adapter_name)
 
+    def add_adapter(self, adapter_name: str, peft_config: PeftConfig) -> None:
+        """
+        Add an adapter to the model based on the passed configuration.
+
+        This adapter is not trained. To load a trained adapter, check out [`PeftModel.load_adapter`].
+
+        The name for the new adapter should be unique.
+
+        The new adapter is not automatically set as the active adapter. Use [`PeftModel.set_adapter`] to set the active
+        adapter.
+
+        Args:
+            adapter_name (`str`):
+                The name of the adapter to be added.
+            peft_config ([`PeftConfig`]):
+                The configuration of the adapter to be added.
+        """
+        # ensure that additional adapters also add the classifier layer to modules_to_save
+        if hasattr(peft_config, "modules_to_save"):
+            classifier_module_names = ["classifier", "score"]
+            if peft_config.modules_to_save is None:
+                peft_config.modules_to_save = classifier_module_names[:]
+            else:
+                peft_config.modules_to_save.extend(classifier_module_names)
+
+        return super().add_adapter(adapter_name, peft_config)
+
     def forward(
         self,
         input_ids=None,
@@ -1675,19 +1710,54 @@ class PeftModelForTokenClassification(PeftModel):
 
     def __init__(self, model: torch.nn.Module, peft_config: PeftConfig = None, adapter_name: str = "default") -> None:
         super().__init__(model, peft_config, adapter_name)
+
+        classifier_module_names = ["classifier", "score"]
         if self.modules_to_save is None:
-            self.modules_to_save = {"classifier", "score"}
+            self.modules_to_save = set(classifier_module_names)
         else:
-            self.modules_to_save.update({"classifier", "score"})
+            self.modules_to_save.update(classifier_module_names)
+
+        if hasattr(peft_config, "modules_to_save"):
+            if peft_config.modules_to_save is None:
+                peft_config.modules_to_save = classifier_module_names[:]
+            else:
+                peft_config.modules_to_save.extend(classifier_module_names)
 
         for name, _ in self.base_model.named_children():
             if any(module_name in name for module_name in self.modules_to_save):
                 self.cls_layer_name = name
                 break
 
-        # to make sure classifier layer is trainable
+        # to make sure classifier layer is trainable; this may add a new ModulesToSaveWrapper
         _set_trainable(self, adapter_name)
 
+    def add_adapter(self, adapter_name: str, peft_config: PeftConfig) -> None:
+        """
+        Add an adapter to the model based on the passed configuration.
+
+        This adapter is not trained. To load a trained adapter, check out [`PeftModel.load_adapter`].
+
+        The name for the new adapter should be unique.
+
+        The new adapter is not automatically set as the active adapter. Use [`PeftModel.set_adapter`] to set the active
+        adapter.
+
+        Args:
+            adapter_name (`str`):
+                The name of the adapter to be added.
+            peft_config ([`PeftConfig`]):
+                The configuration of the adapter to be added.
+        """
+        # ensure that additional adapters also add the classifier layer to modules_to_save
+        if hasattr(peft_config, "modules_to_save"):
+            classifier_module_names = ["classifier", "score"]
+            if peft_config.modules_to_save is None:
+                peft_config.modules_to_save = classifier_module_names[:]
+            else:
+                peft_config.modules_to_save.extend(classifier_module_names)
+
+        return super().add_adapter(adapter_name, peft_config)
+
     def forward(
         self,
         input_ids=None,
@@ -1850,19 +1920,54 @@ class PeftModelForQuestionAnswering(PeftModel):
 
     def __init__(self, model: torch.nn.Module, peft_config: PeftConfig, adapter_name: str = "default") -> None:
         super().__init__(model, peft_config, adapter_name)
+
+        qa_module_names = ["qa_outputs"]
         if self.modules_to_save is None:
-            self.modules_to_save = {"qa_outputs"}
+            self.modules_to_save = set(qa_module_names)
         else:
-            self.modules_to_save.update({"qa_outputs"})
+            self.modules_to_save.update(qa_module_names)
+
+        if hasattr(peft_config, "modules_to_save"):
+            if peft_config.modules_to_save is None:
+                peft_config.modules_to_save = qa_module_names[:]
+            else:
+                peft_config.modules_to_save.extend(qa_module_names)
 
         for name, _ in self.base_model.named_children():
             if any(module_name in name for module_name in self.modules_to_save):
                 self.cls_layer_name = name
                 break
 
-        # to make sure classifier layer is trainable
+        # to make sure classifier layer is trainable; this may add a new ModulesToSaveWrapper
         _set_trainable(self, adapter_name)
 
+    def add_adapter(self, adapter_name: str, peft_config: PeftConfig) -> None:
+        """
+        Add an adapter to the model based on the passed configuration.
+
+        This adapter is not trained. To load a trained adapter, check out [`PeftModel.load_adapter`].
+
+        The name for the new adapter should be unique.
+
+        The new adapter is not automatically set as the active adapter. Use [`PeftModel.set_adapter`] to set the active
+        adapter.
+
+        Args:
+            adapter_name (`str`):
+                The name of the adapter to be added.
+            peft_config ([`PeftConfig`]):
+                The configuration of the adapter to be added.
+        """
+        # ensure that additional adapters also add the classifier layer to modules_to_save
+        if hasattr(peft_config, "modules_to_save"):
+            qa_module_names = ["qa_outputs"]
+            if peft_config.modules_to_save is None:
+                peft_config.modules_to_save = qa_module_names[:]
+            else:
+                peft_config.modules_to_save.extend(qa_module_names)
+
+        return super().add_adapter(adapter_name, peft_config)
+
     def forward(
         self,
         input_ids=None,
diff --git a/src/peft/tuners/lora/model.py b/src/peft/tuners/lora/model.py
--- a/src/peft/tuners/lora/model.py
+++ b/src/peft/tuners/lora/model.py
@@ -449,17 +449,85 @@ def _unload_and_optionally_merge(
 
         return self.model
 
+    def _check_add_weighted_adapter(
+        self, adapters: list[str], combination_type: str, svd_rank: int | None
+    ) -> tuple[str, int, str]:
+        """
+        Helper function to check if the arguments to add_weighted_adapter are valid and compatible with the underlying
+        model.
+        """
+        for adapter in adapters:
+            if adapter not in list(self.peft_config.keys()):
+                raise ValueError(f"Adapter {adapter} does not exist")
+
+        # If more than one of the adapters targets the same module with modules_to_save, raise an error, as these
+        # modules cannot be merged. First, find the ModulesToSaveWrapper instances in the model, then check if they
+        # have modules for the adapters to be merged.
+        modules_to_save_wrappers = [module for module in self.modules() if isinstance(module, ModulesToSaveWrapper)]
+        problematic_wrappers = [
+            wrapper
+            for wrapper in modules_to_save_wrappers
+            if sum(adapter in wrapper.modules_to_save for adapter in adapters) > 1
+        ]
+        if problematic_wrappers:
+            raise ValueError(
+                "Cannot add weighted adapters if they target the same module with modules_to_save, but found "
+                f"{len(problematic_wrappers)} such instance(s)."
+            )
+
+        # if there is only one adapter, we can only use linear merging
+        combination_type = "linear" if len(adapters) == 1 else combination_type
+
+        adapters_ranks = [self.peft_config[adapter].r for adapter in adapters]
+        if combination_type in ("linear", "ties", "dare_ties", "dare_linear", "magnitude_prune"):
+            # all adapters ranks should be same, new rank is just this value
+            if len(set(adapters_ranks)) != 1:
+                raise ValueError(
+                    "All adapters must have the same r value when using combination_type linear, ties, dare_ties or "
+                    "dare_linear."
+                )
+            new_rank = adapters_ranks[0]
+        elif combination_type == "cat":
+            # adapters ranks may be different, new rank is sum of all ranks
+            # be careful, because output adapter rank may be really big if mixing a lot of adapters
+            new_rank = sum(adapters_ranks)
+        elif combination_type.endswith("svd"):
+            # new rank is the max of all ranks of the adapters if not provided
+            new_rank = svd_rank or max(adapters_ranks)
+        else:
+            raise ValueError(f"Invalid combination_type: {combination_type}")
+
+        target_module_types = [type(self.peft_config[adapter].target_modules) for adapter in adapters]
+        if not target_module_types:
+            raise ValueError(f"Found no adapter matching the names in {adapters}")
+        if len(set(target_module_types)) > 1:
+            raise ValueError(
+                "all adapter configs should follow the same target modules type. "
+                "Combining adapters with `target_modules` type being a mix of list/set and string is not supported."
+            )
+
+        if target_module_types[0] == str:
+            new_target_modules = "|".join(f"({self.peft_config[adapter].target_modules})" for adapter in adapters)
+        elif target_module_types[0] == set:
+            new_target_modules = reduce(
+                operator.or_, (self.peft_config[adapter].target_modules for adapter in adapters)
+            )
+        else:
+            raise TypeError(f"Invalid type {target_module_types[0]} found in target_modules")
+
+        return combination_type, new_rank, new_target_modules
+
     def add_weighted_adapter(
         self,
-        adapters,
-        weights,
-        adapter_name,
-        combination_type="svd",
-        svd_rank=None,
-        svd_clamp=None,
-        svd_full_matrices=True,
-        svd_driver=None,
-        density=None,
+        adapters: list[str],
+        weights: list[float],
+        adapter_name: str,
+        combination_type: str = "svd",
+        svd_rank: int | None = None,
+        svd_clamp: int | None = None,
+        svd_full_matrices: bool = True,
+        svd_driver: str | None = None,
+        density: float | None = None,
         majority_sign_method: Literal["total", "frequency"] = "total",
     ) -> None:
         """
@@ -508,44 +576,11 @@ def add_weighted_adapter(
             if adapter not in list(self.peft_config.keys()):
                 raise ValueError(f"Adapter {adapter} does not exist")
 
-        # if there is only one adapter, we can only use linear merging
-        combination_type = "linear" if len(adapters) == 1 else combination_type
-
-        adapters_ranks = [self.peft_config[adapter].r for adapter in adapters]
-        if combination_type in ("linear", "ties", "dare_ties", "dare_linear", "magnitude_prune"):
-            # all adapters ranks should be same, new rank is just this value
-            if len(set(adapters_ranks)) != 1:
-                raise ValueError(
-                    "All adapters must have the same r value when using combination_type linear, ties, dare_ties or dare_linear."
-                )
-            new_rank = adapters_ranks[0]
-        elif combination_type == "cat":
-            # adapters ranks may be different, new rank is sum of all ranks
-            # be careful, because output adapter rank may be really big if mixing a lot of adapters
-            new_rank = sum(adapters_ranks)
-        elif combination_type.endswith("svd"):
-            # new rank is the max of all ranks of the adapters if not provided
-            new_rank = svd_rank or max(adapters_ranks)
-        else:
-            raise ValueError(f"Invalid combination_type: {combination_type}")
-
-        target_module_types = [type(self.peft_config[adapter].target_modules) for adapter in adapters]
-        if not target_module_types:
-            raise ValueError(f"Found no adapter matching the names in {adapters}")
-        if len(set(target_module_types)) > 1:
-            raise ValueError(
-                "all adapter configs should follow the same target modules type. "
-                "Combining adapters with `target_modules` type being a mix of list/set and string is not supported."
-            )
-
-        if target_module_types[0] == str:
-            new_target_modules = "|".join(f"({self.peft_config[adapter].target_modules})" for adapter in adapters)
-        elif target_module_types[0] == set:
-            new_target_modules = reduce(
-                operator.or_, (self.peft_config[adapter].target_modules for adapter in adapters)
-            )
-        else:
-            raise TypeError(f"Invalid type {target_module_types[0]} found in target_modules")
+        combination_type, new_rank, new_target_modules = self._check_add_weighted_adapter(
+            adapters=adapters,
+            combination_type=combination_type,
+            svd_rank=svd_rank,
+        )
 
         self.peft_config[adapter_name] = replace(
             self.peft_config[adapters[0]],
diff --git a/src/peft/utils/other.py b/src/peft/utils/other.py
--- a/src/peft/utils/other.py
+++ b/src/peft/utils/other.py
@@ -331,7 +331,13 @@ def check_adapter_name(adapter_name):
         if isinstance(module, ModulesToSaveWrapper):
             # only check the adapter_name if we actually encounter a ModulesToSaveWrapper, otherwise we don't care
             adapter_name = check_adapter_name(adapter_name)
-            module.set_adapter(adapter_name)
+
+            # if the adapter is found in this module, set it as the active adapter, else disable the adapters of this
+            # module
+            if adapter_name in module.modules_to_save:
+                module.set_adapter(adapter_name)
+            else:
+                module.enable_adapters(False)
 
 
 def _prepare_prompt_learning_config(peft_config, model_config):


Test patch:
diff --git a/tests/test_custom_models.py b/tests/test_custom_models.py
--- a/tests/test_custom_models.py
+++ b/tests/test_custom_models.py
@@ -16,20 +16,32 @@
 # limitations under the License.
 import copy
 import os
+import re
 import shutil
 import tempfile
 import time
 import unittest
 from contextlib import contextmanager
+from functools import partial
 
 import pytest
 import torch
 from parameterized import parameterized
 from torch import nn
-from transformers import AutoModelForCausalLM
+from transformers import AutoModelForCausalLM, AutoModelForSequenceClassification
 from transformers.pytorch_utils import Conv1D
 
-from peft import AdaLoraConfig, IA3Config, LoHaConfig, LoKrConfig, LoraConfig, OFTConfig, PeftModel, get_peft_model
+from peft import (
+    AdaLoraConfig,
+    IA3Config,
+    LoHaConfig,
+    LoKrConfig,
+    LoraConfig,
+    OFTConfig,
+    PeftModel,
+    TaskType,
+    get_peft_model,
+)
 from peft.tuners.tuners_utils import BaseTunerLayer
 from peft.utils import ModulesToSaveWrapper, infer_device
 
@@ -819,6 +831,131 @@ def test_weight_bias_attributes(self):
         assert hasattr(model.base_model.model.lin0, "weight")
         assert hasattr(model.base_model.model.lin0, "bias")
 
+    def test_multiple_adapters_automatic_modules_to_save(self):
+        # See issue 1574
+        # When we use certain task types, PeftModel.modules_to_save is automatically updated to include some extra
+        # layers not specified in the PeftConfig. This attribute should be honored for all adapters, not just for
+        # the default adapter.
+        config0 = LoraConfig(task_type=TaskType.SEQ_CLS)
+        config1 = LoraConfig(task_type=TaskType.SEQ_CLS)
+        model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased")
+        model = get_peft_model(model, config0)
+        # sanity check
+        assert model.modules_to_save
+
+        model.add_adapter("other", config1)
+        assert "default" in model.base_model.classifier.modules_to_save
+        assert "other" in model.base_model.classifier.modules_to_save
+
+    @parameterized.expand([IA3Config, LoHaConfig, LoKrConfig, LoraConfig, OFTConfig])
+    def test_multiple_adapters_mixed_modules_to_save(self, config_cls):
+        # See issue 1574
+        # Check that we can have a model where one adapter has modules_to_save and the other doesn't. It should be
+        # possible to switch between those adapters and to use them.
+        if hasattr(config_cls, "feedforward_modules"):  # IA³
+            config_cls = partial(config_cls, feedforward_modules=["lin0"])
+
+        config0 = config_cls(target_modules=["lin0"], modules_to_save=["lin1"])
+        config1 = config_cls(target_modules=["lin0"])
+        model = MLP()
+        model = get_peft_model(model, config0).to(self.torch_device)
+        model.add_adapter("other", config1)
+
+        assert "default" in model.base_model.lin1.modules_to_save
+        assert "other" not in model.base_model.lin1.modules_to_save
+
+        # check that switching adapters and predicting does not raise
+        inputs = self.prepare_inputs_for_testing()
+        # "default" adapter is active
+        model(**inputs)
+        # switch to "other" adapter
+        model.set_adapter("other")
+        model(**inputs)
+
+    @parameterized.expand([IA3Config, LoHaConfig, LoKrConfig, LoraConfig, OFTConfig])
+    def test_multiple_adapters_mixed_modules_to_save_order_switched(self, config_cls):
+        # See issue 1574
+        # Same test as test_multiple_adapters_mixed_modules_to_save, but this time the 2nd adapter has modules_to_save.
+        if hasattr(config_cls, "feedforward_modules"):  # IA³
+            config_cls = partial(config_cls, feedforward_modules=["lin0"])
+
+        config0 = config_cls(target_modules=["lin0"])
+        config1 = config_cls(target_modules=["lin0"], modules_to_save=["lin1"])
+        model = MLP()
+        model = get_peft_model(model, config0).to(self.torch_device)
+        model.add_adapter("other", config1)
+
+        assert "default" not in model.base_model.lin1.modules_to_save
+        assert "other" in model.base_model.lin1.modules_to_save
+
+        # check that switching adapters and predicting does not raise
+        inputs = self.prepare_inputs_for_testing()
+        # "default" adapter is active
+        model(**inputs)
+        # switch to "other" adapter
+        model.set_adapter("other")
+        model(**inputs)
+
+    def test_multiple_adapters_mixed_modules_to_save_merging_adapters(self):
+        # See issue 1574
+        # This test is similar to test_multiple_adapters_mixed_modules_to_save, but it also checks that merging adapter
+        # weights works when one adapter has a modules_to_save and the other hasn't
+        config0 = LoraConfig(target_modules=["lin0"], modules_to_save=["lin1"])
+        config1 = LoraConfig(target_modules=["lin0"])
+        model = MLP()
+        model = get_peft_model(model, config0).to(self.torch_device)
+        model.add_adapter("other", config1)
+
+        # check that this does not raise
+        model.add_weighted_adapter(["default", "other"], weights=[1.0, 1.0], adapter_name="merged")
+
+        # since one of the adapters that was merged has a modules_to_save, that one should be used for the merged
+        # adapter
+        assert "default" in model.base_model.model.lin1.modules_to_save
+        assert "other" not in model.base_model.model.lin1.modules_to_save
+        assert "merged" in model.base_model.model.lin1.modules_to_save
+
+        # check that using the merged adapter does not raise
+        model.set_adapter("merged")
+        inputs = self.prepare_inputs_for_testing()
+        model(**inputs)
+
+    def test_multiple_adapters_same_modules_to_save_merging_adapters_raises(self):
+        # See issue 1574
+        # This test is similar to test_multiple_adapters_mixed_modules_to_save_merging_adapters but here the two
+        # adapters target the same module with modules_to_save. In this case, trying to merge the adapter weights
+        # should raise an error.
+        config0 = LoraConfig(target_modules=["lin0"], modules_to_save=["lin1"])
+        config1 = LoraConfig(target_modules=["lin0"], modules_to_save=["lin1"])
+        model = MLP()
+        model = get_peft_model(model, config0).to(self.torch_device)
+        model.add_adapter("other", config1)
+
+        msg = re.escape(
+            "Cannot add weighted adapters if they target the same module with modules_to_save, but found 1 such "
+            "instance(s)."
+        )
+        with pytest.raises(ValueError, match=msg):
+            model.add_weighted_adapter(["default", "other"], weights=[1.0, 1.0], adapter_name="merged")
+
+    def test_multiple_adapters_seq_cls_mixed_modules_to_save_merging_adapters(self):
+        # See issue 1574
+        # This test is similar to test_multiple_adapters_mixed_modules_to_save_merging_adapters but uses a SEQ_CLS
+        # model like in test_multiple_adapters_automatic_modules_to_save. This should raise an error because the same
+        # module is implicitly targeted by modules_to_save twice.
+        config0 = LoraConfig(task_type=TaskType.SEQ_CLS)
+        config1 = LoraConfig(task_type=TaskType.SEQ_CLS)
+        model = AutoModelForSequenceClassification.from_pretrained("bert-base-uncased")
+        model = get_peft_model(model, config0)
+        model.add_adapter("other", config1)
+
+        msg = re.escape(
+            "Cannot add weighted adapters if they target the same module with modules_to_save, but found 1 such "
+            "instance(s)."
+        )
+        with pytest.raises(ValueError, match=msg):
+            model.add_weighted_adapter(["default", "other"], weights=[1.0, 1.0], adapter_name="merged")
+
     def test_existing_model_card(self):
         # ensure that if there is already a model card, it is not overwritten
         model = MLP()