Repository: huggingface/transformers
Pull request: FIX Error when trying to load non-LoRA PEFT

# What does this PR do?

This PR fixes a bug that prevented non-LoRA PEFT adapters to be loaded into a transformers model. It was just a very simple logical error where a check was performed generally when it actually only is relevant for hotswapping. A test for this has been added.

Additionally, also testing if a non-LoRA adapter can be added to a transformers model. This was not broken but still lacked test coverage.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

Changed files:
filename: src/transformers/integrations/peft.py
patch: @@ -279,9 +279,6 @@ def load_adapter(
             )
             peft_config.inference_mode = not is_trainable
 
-        if peft_config.peft_type != PeftType.LORA:
-            raise ValueError("Hotswapping is currently only supported for LoRA, please set `hotswap=False`.")
-
         if not hotswap:
             # TODO: WE NEED TOO APPLY OUR DYNAMIC WEIGHT CONVERSION AT SOME POINT HERE!
             # Create and add fresh new adapters into the model, unless the weights are hotswapped

filename: tests/peft_integration/test_peft_integration.py
patch: @@ -889,6 +889,60 @@ def test_peft_pipeline_no_warning(self):
             # Generate text to verify pipeline works
             _ = lora_generator(text, max_new_tokens=20)
 
+    def test_non_lora_load_adapter(self):
+        """
+        Check that loading a non-LoRA adapter works. Using LoKr as an example, not testing all possible PEFT methods.
+        """
+        from peft import LoKrConfig, get_peft_model
+
+        inputs = torch.randint(0, 100, (1, 10)).to(torch_device)
+        atol, rtol = 1e-4, 1e-4
+
+        for model_id in self.transformers_test_model_ids:
+            for transformers_class in self.transformers_test_model_classes:
+                model = transformers_class.from_pretrained(model_id).to(torch_device)
+                with torch.inference_mode():
+                    output_base = model(inputs).logits
+
+                peft_config = LoKrConfig(init_weights=False)
+                peft_model = get_peft_model(model, peft_config)
+                with torch.inference_mode():
+                    output_peft = peft_model(inputs).logits
+
+                # sanity check: should be different
+                assert not torch.allclose(output_base, output_peft, atol=atol, rtol=rtol)
+
+                with tempfile.TemporaryDirectory() as tmpdirname:
+                    peft_model.save_pretrained(tmpdirname)
+                    del model, peft_model
+
+                    model = transformers_class.from_pretrained(tmpdirname).to(torch_device)
+                    with torch.inference_mode():
+                        output_transformers = model(inputs).logits
+                    self.assertTrue(torch.allclose(output_peft, output_transformers, atol=atol, rtol=rtol))
+
+    def test_non_lora_add_adapter(self):
+        """
+        Check that adding a non-LoRA adapter works. Using LoKr as an example, not testing all possible PEFT methods.
+        """
+        from peft import LoKrConfig
+
+        inputs = torch.randint(0, 100, (1, 10)).to(torch_device)
+        atol, rtol = 1e-4, 1e-4
+
+        for model_id in self.transformers_test_model_ids:
+            for transformers_class in self.transformers_test_model_classes:
+                model = transformers_class.from_pretrained(model_id).to(torch_device)
+                with torch.inference_mode():
+                    output_base = model(inputs).logits
+
+                peft_config = LoKrConfig(init_weights=False)
+                model.add_adapter(peft_config)
+                with torch.inference_mode():
+                    output_peft = model(inputs).logits
+                # should be different
+                assert not torch.allclose(output_base, output_peft, atol=atol, rtol=rtol)
+
 
 @require_peft
 @require_torch