Repository: huggingface/peft

Issue:
Is there a choice to use `ignore_mismatch_sizes` in PeftModel.from_pretrained like AutoModel in Transformers?
### Feature request

Add ignore_mismatch_sizes in PeftModel.from_pretrained.

### Motivation

When I try to add some special tokens into an LM and use an adapter, I find though I can set ignore_mismatch_sizes = True when I load the base model, I have no similar choice to ignore the mismatch parameters when I load the adapter.
I have searched a possible way to first load the adapter and then add the base model. However, in my case, my base model has been some trained parameters that I cannot resize the parameters after loading the adapter.
Have I miss some args? Or is it a necessary choice?

### Your contribution

I don't know what can I do.


Discussion:
Do you have some simple reproducer for the error you mentioned? Then we can check if there is something we can do on the PEFT side.

From what I understand, this shouldn't require a trained adapter, so it you could just show how you created and saved the adapter, then tried to load it, it should be sufficient.
@BenjaminBossan Sorry, my statement was unclear. I did not encounter a bug, but I wish to submit a **feature request**. When I try to add new tokens to a pre-trained model **as well as** a corresponding adapter (which means the size of lm_head and token embed needs to change), I find that the `from_pretrained` function in transformers can conveniently ignore those modules with the same name but mismatched size by using the ignore_mismatch_size option. (See [here](https://huggingface.co/docs/transformers/v4.39.3/en/main_classes/model#transformers.PreTrainedModel.from_pretrained))

This is useful when I want to further fine-tune these models. However, I noticed that peft does not offer this option, meaning I cannot load a partially matched adapter into a base model that has loaded some parameters through the above method (with another part of the parameters being reinitialized due to size change). I hope Peft could offer a similar functionality.

In my case, I have trained a Llama with an adapter on my task, I use the `trainer.train` to automatically save the adapter, and the adapter has been saved successfully. But I want to use this model as well as the trained adpater to finetune on another task, which requires two special tokens, so I resize the base model (here, it means the Llama), and I can successfully load the pretrained Llama (except for the resized lm_head and embed_layer, which I will manually process them). However, I find I cannot load the adapter because I have changed the size of these two modules. I hope I have an option to just ignore them.
I forgot to mention that for these two layers, I set them in `modules_to_save`, so they are both fully trained and saved in the adapter.
Thanks you for clarifying, I think I got it now. So you would like something like this to work:

```python
import torch
from transformers import AutoModelForCausalLM
from peft import LoraConfig, PeftModel, get_peft_model
from peft.utils.other import ModulesToSaveWrapper

model_id = "facebook/opt-125m"
model0 = AutoModelForCausalLM.from_pretrained(model_id)
peft_config0 = LoraConfig(
    task_type="CAUSAL_LM", target_modules=["q_proj", "v_proj"], modules_to_save=["embed_tokens"], init_lora_weights=False
)
peft_model0 = get_peft_model(model0, peft_config0)

# embedding layer uses ModulesToSave:
[module for module in peft_model0.modules() if isinstance(module, ModulesToSaveWrapper)]

peft_model0.save_pretrained("/tmp/peft/issue-1605")

model1 = AutoModelForCausalLM.from_pretrained(model_id)
model1.resize_token_embeddings(50272 + 2)
# this currently raises an error:
peft_model1 = PeftModel.from_pretrained(model1, "/tmp/peft/issue-1605", ignore_mismatched_sizes=True)
# RuntimeError: Error(s) in loading state_dict for PeftModelForCausalLM:
# 	size mismatch for base_model.model.model.decoder.embed_tokens.modules_to_save.default.weight: copying a param with shape torch.Size([50272, 768]) from checkpoint, the shape in current model is torch.Size([50274, 768]).
```

Is that right?
@BenjaminBossan Yes! That is right!
In fact, I am currently circumventing this issue by creating a new peft based on the new model (by get_peft_model) and then manually extracting parameters from safetensors. I believe providing an option to directly ignore mismatched parameters could be helpful.

Resolving patch:
diff --git a/src/peft/peft_model.py b/src/peft/peft_model.py
--- a/src/peft/peft_model.py
+++ b/src/peft/peft_model.py
@@ -839,7 +839,10 @@ def load_adapter(self, model_id: str, adapter_name: str, is_trainable: bool = Fa
         adapters_weights = load_peft_weights(model_id, device=torch_device, **hf_hub_download_kwargs)
 
         # load the weights into the model
-        load_result = set_peft_model_state_dict(self, adapters_weights, adapter_name=adapter_name)
+        ignore_mismatched_sizes = kwargs.get("ignore_mismatched_sizes", False)
+        load_result = set_peft_model_state_dict(
+            self, adapters_weights, adapter_name=adapter_name, ignore_mismatched_sizes=ignore_mismatched_sizes
+        )
         if (
             (getattr(self, "hf_device_map", None) is not None)
             and (len(set(self.hf_device_map.values()).intersection({"cpu", "disk"})) > 0)
diff --git a/src/peft/utils/save_and_load.py b/src/peft/utils/save_and_load.py
--- a/src/peft/utils/save_and_load.py
+++ b/src/peft/utils/save_and_load.py
@@ -11,6 +11,8 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+from __future__ import annotations
+
 import os
 import warnings
 from typing import Optional
@@ -186,13 +188,49 @@ def get_peft_model_state_dict(
     return to_return
 
 
-def set_peft_model_state_dict(model, peft_model_state_dict, adapter_name="default"):
+def _find_mismatched_keys(
+    model: torch.nn.Module, peft_model_state_dict: dict[str, torch.Tensor], ignore_mismatched_sizes: bool = False
+) -> tuple[dict[str, torch.Tensor], list[tuple[str, tuple[int, ...], tuple[int, ...]]]]:
+    if not ignore_mismatched_sizes:
+        return peft_model_state_dict, []
+
+    mismatched = []
+    state_dict = model.state_dict()
+    for key, tensor in peft_model_state_dict.items():
+        if key not in state_dict:
+            continue
+
+        # see https://github.com/huggingface/transformers/blob/09f9f566de83eef1f13ee83b5a1bbeebde5c80c1/src/transformers/modeling_utils.py#L3858-L3864
+        if (state_dict[key].shape[-1] == 1) and (state_dict[key].numel() * 2 == tensor.numel()):
+            # This skips size mismatches for 4-bit weights. Two 4-bit values share an 8-bit container, causing size
+            # differences. Without matching with module type or paramter type it seems like a practical way to detect
+            # valid 4bit weights.
+            continue
+
+        if state_dict[key].shape != tensor.shape:
+            mismatched.append((key, tensor.shape, state_dict[key].shape))
+
+    for key, _, _ in mismatched:
+        del peft_model_state_dict[key]
+
+    return peft_model_state_dict, mismatched
+
+
+def set_peft_model_state_dict(
+    model, peft_model_state_dict, adapter_name="default", ignore_mismatched_sizes: bool = False
+):
     """
     Set the state dict of the Peft model.
 
     Args:
-        model ([`PeftModel`]): The Peft model.
-        peft_model_state_dict (`dict`): The state dict of the Peft model.
+        model ([`PeftModel`]):
+            The Peft model.
+        peft_model_state_dict (`dict`):
+            The state dict of the Peft model.
+        adapter_name (`str`, *optional*, defaults to `"default"`):
+            The name of the adapter whose state dict should be set.
+        ignore_mismatched_sizes (`bool`, *optional*, defaults to `False`):
+            Whether to ignore mismatched in the state dict.
     """
     config = model.peft_config[adapter_name]
     state_dict = {}
@@ -246,6 +284,9 @@ def set_peft_model_state_dict(model, peft_model_state_dict, adapter_name="defaul
     else:
         raise NotImplementedError
 
+    peft_model_state_dict, mismatched_keys = _find_mismatched_keys(
+        model, peft_model_state_dict, ignore_mismatched_sizes=ignore_mismatched_sizes
+    )
     load_result = model.load_state_dict(peft_model_state_dict, strict=False)
     if config.is_prompt_learning:
         model.prompt_encoder[adapter_name].embedding.load_state_dict(
@@ -254,6 +295,20 @@ def set_peft_model_state_dict(model, peft_model_state_dict, adapter_name="defaul
 
     if config.peft_type == PeftType.MULTITASK_PROMPT_TUNING:
         model.prompt_encoder[adapter_name].load_state_dict(peft_model_state_dict, strict=False)
+
+    if mismatched_keys:
+        # see https://github.com/huggingface/transformers/blob/09f9f566de83eef1f13ee83b5a1bbeebde5c80c1/src/transformers/modeling_utils.py#L4039
+        mismatched_warning = "\n".join(
+            [
+                f"- {key}: found shape {shape1} in the checkpoint and {shape2} in the model instantiated"
+                for key, shape1, shape2 in mismatched_keys
+            ]
+        )
+        msg = (
+            f"Some weights of {model.__class__.__name__} were not initialized from the model checkpoint "
+            f"and are being ignored because you passed `ignore_mismatched_sizes=True`: {mismatched_warning}."
+        )
+        warnings.warn(msg)
     return load_result
 
 


Test patch:
diff --git a/tests/test_custom_models.py b/tests/test_custom_models.py
--- a/tests/test_custom_models.py
+++ b/tests/test_custom_models.py
@@ -16,6 +16,7 @@
 # limitations under the License.
 import copy
 import os
+import shutil
 import tempfile
 import time
 import unittest
@@ -352,9 +353,9 @@ def forward(self, X):
 
 
 class ModelEmbConv1D(nn.Module):
-    def __init__(self):
+    def __init__(self, emb_size=100):
         super().__init__()
-        self.emb = nn.Embedding(100, 5)
+        self.emb = nn.Embedding(emb_size, 5)
         self.conv1d = Conv1D(1, 5)
         self.relu = nn.ReLU()
         self.flat = nn.Flatten()
@@ -901,6 +902,35 @@ def test_targeting_lora_to_embedding_layer_non_transformers(self, save_embedding
             assert "base_model.model.emb.base_layer.weight" not in state_dict
             del state_dict
 
+    def test_load_resized_embedding_ignore_mismatched_sizes(self):
+        # issue #1605
+        # Make it possible to load a LoRA layer that targets an embedding layer even if the sizes mismatch by passing
+        # ignore_mismatched_sizes=True
+        model = ModelEmbConv1D(emb_size=100)
+        config = LoraConfig(target_modules=["emb", "lin0"], init_lora_weights=False)
+        model = get_peft_model(model, config)
+
+        # note: not using the context manager here because it fails on Windows CI for some reason
+        tmp_dirname = tempfile.mkdtemp()
+        try:
+            model.save_pretrained(tmp_dirname)
+            model = ModelEmbConv1D(emb_size=105)
+
+            # first check that this raises
+            with pytest.raises(RuntimeError) as exc:
+                PeftModel.from_pretrained(model, tmp_dirname)
+            msg = exc.value.args[0]
+            assert "size mismatch" in msg and "100" in msg and "105" in msg
+
+            # does not raise
+            PeftModel.from_pretrained(model, tmp_dirname, ignore_mismatched_sizes=True)
+        finally:
+            try:
+                shutil.rmtree(tmp_dirname)
+            except PermissionError:
+                # windows error
+                pass
+
     @parameterized.expand(
         [
             LoraConfig(target_modules=["lin0"], init_lora_weights=False),