FIX Restore LoRA hotswapping functionality
Author: BenjaminBossan
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=45682)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=45682)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

LoRA hotswapping was added in #41297. Due to changes in #43261, it stopped working. This PR restores the functionality.

The tests already cover this and are failing, but probably no one noticed because they're slow tests. On main, they fail with mismatched sizes, which is expected as the padding of the LoRA weights is not being applied. With this PR, I can confirm that the tests pass locally.

Since the two PRs were released in together in v5, there was never a Transformers release with working hotswapping functionality.

### Notes

The hotswap path does not use `_load_pretrained_model`, which means that loading the `state_dict` if not present is required. I hoisted that functionality from the TP path, which was already there, to re-use the same logic. I also apply weight renamings for that reason.

Moreover, I moved the inference model logic to a local function, again to avoid duplicating the logic.

## Code Agent Policy

- [x] I confirm that this is not a pure code agent PR. Claude was used to assist with writing this PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45682). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@BenjaminBossan Thanks for the fix.

1. Do you know whether text encoder hotswapping is something that’s planned, either in Transformers/PEFT or via a corresponding diffusers change? 
2. IIs there a workaround I can use for LoRAs (SDXL) that include text encoder weights?

> 1. Do you know whether text encoder hotswapping is something that’s planned, either in Transformers/PEFT or via a corresponding diffusers change?

If this is merged, it should work directly in Transformers.

> 2\. IIs there a workaround I can use for LoRAs (SDXL) that include text encoder weights?

Hotswapping is already possible in PEFT itself: https://huggingface.co/docs/peft/v0.19.0/en/package_reference/hotswap. This PR is purely about providing the functionality directly in the PEFT-Transformers integration.

I was confused why I couldn't hotswap LoRAs that have text encoders when I use diffusers which uses PEFT under the hood. If I call:
```
pipeline.load_lora_weights(lora_file_path, adapter_name=lora_name, hotswap=True)
```
Then I get the error:
```
exception=ValueError('At the moment, hotswapping is not supported for text encoders, please pass `hotswap=False`.')>
```
This is indicated in the docs as well: 
```
Hotswapping is unsupported for LoRAs that target the text encoder.
```

I can open a new issue or discussion. I don't want to pollute this PR with unrelated comments.

> I can open a new issue or discussion. I don't want to pollute this PR with unrelated comments.

No need, we are aware and we need to enable hotswapping in Transformers first, as Diffusers uses Transformers for the text models.

Also my main comment is that this code should be inside peft directly and the bridge layer should be as small as possible. 

AFAICT, failing CI is unrelated to this PR.

@ArthurZucker I merged the latest main and resolved a merge conflict. The failing CI seems to be unrelated:

- `tests/models/glm_moe_dsa/test_modeling_glm_moe_dsa.py::GlmMoeDsaModelTest::test_tp_backward`
- `tests/models/olmo3/test_modeling_olmo3.py::Olmo3ModelTest::test_tp_generation_quantized`
- `tests/models/mistral/test_modeling_mistral.py::MistralModelTest::test_tp_generation`

Thanks for the review @ArthurZucker. I indeed misunderstood `rename_source_key` and now the weight conversion logic should be properly reimplemented for the hotswap path. I also found an issue with the mixtral test not working with `torch.compile(..., mode="reduce-overhead")` that I fixed.

> but this does not work if you swap mixtral for mixtral -> its missing that test with non v5 ckpt

Could you please share how you tested that? Hotswapping currently does not work with MoE layers (as they require targeting the `nn.Parameter` directly, which needs to be added in PEFT) so I think this is not working no matter what.

Failing CI seems to be unrelated:

- `tests/models/kosmos2_5/test_modeling_kosmos2_5.py::Kosmos2_5ModelTest::test_generate_with_cache_matches_no_cache`
- `tests/models/gpt_oss/test_modeling_gpt_oss.py::GptOssModelTest::test_ep_forward`

I did not test I was more saying if you have any weight conversion ops -> mixtral does and is a simple case! 
Ok having a look

> I did not test I was more saying if you have any weight conversion ops -> mixtral does and is a simple case!

Ah okay, but I think even with correct conversion, it would not work for the reason mentioned above. I think it's enough of an edge case (weight conversion + hotswap + no MoE) that we can ignore it for now.

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45682&sha=8211bb

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=45682)
**Latest run:** [28794746208:1](https://github.com/huggingface/transformers/actions/runs/28794746208)
**Result:** `failure` | **Jobs:** 15 | **Tests:** 171,024 | **Failures:** 6 | **Duration:** 21h 12m

<!-- ci-dashboard-recap:end -->

@ArthurZucker I brought the PR up-to-date with the latest Transformers changes. I also tackled the issue of whether hotswapping can work with v4->v5 weight conversion. Turns out that there wasn't really any blocker from the PEFT side, just some minor changes needed on Transformers side, which I added to the PR. New tests cover this case. (Minor note: Targeting `nn.Parameter`s with `torch.compile` enabled *will* trigger re-compilation. But that's not directly related to this PR, it's a fundamental issue with using `nn.utils.parametrize`, which is the PEFT mechanism for targeting `nn.Parameter`s.)

PEFT tests are now part of the Transformers CI and passed, but the hotswap tests are slow tests, so I ran them locally with GPU to confirm that they pass. Also some tests are guarded behind PEFT >= v0.20.0, I commented those guards out and ensured they pass with PEFT from main.

The failing CI tests seem to be unrelated to this PR.