Trainer._load_from_checkpoint - support loading multiple Peft adapters
Author: claralp
Pull request: yes

# What does this PR do?

Since it it possible to have multiple Peft adapters in the same model, it should also be possible to resume a training of such models from checkpoint with [transformers.Trainer.train(resume_from_checkpoint=True|"path")](https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Trainer.train).  
No documentation changes because this is a fix of sth already existing, tested with DPO/KTO

Fixes #30478



## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@lewtun @kashif @younesbelkada 


Comments:
nice @claralp would you mind also adding some tests?

@kashif will try. Guess I can use a similar model setup as here [test_trainer.py#L933](https://github.com/huggingface/transformers/blob/main/tests/trainer/test_trainer.py#L933)  

Another question: How is the checkpoint loading with DeepSpeed supposed to work with this [trainer.py#L1847](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L1847)?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30505). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@kashif @lewtun please check if this works now for you.

Note: Discovered another bug while implementing this.
While the `PeftAdapterMixin.set_adapter` function supports multiple active adapters, its internal logic for setting multiple adapters active is broken.  
But this needs to be fixed in the Peft library first 

ok @claralp let's link the peft issue here too then?

@kashif, @younesbelkada the issue cannot be reproduced anymore.   

The background: 
Since multiple adapters can be active at the same time I would have expected the same in [src/peft/utils/other.py#L272](https://github.com/huggingface/peft/blob/main/src/peft/utils/other.py#L272).
I had an error previously when calling this with a list of adapters in [src/transformers/integrations/peft.py#L311](https://github.com/huggingface/transformers/blob/main/src/transformers/integrations/peft.py#L311).  
This was before [commits/16bd0ef66f642ef4b69f4f21cfc368d548318672](https://github.com/huggingface/transformers/pull/30505/commits/16bd0ef66f642ef4b69f4f21cfc368d548318672) when I accessed the `active_adapters` property instead of `active_adapter`.  
But when I now run a test case with two active adapters, it works.  
So I would treat this as solved

@kashif @younesbelkada @lewtun is there any open point for me to fix now or is this ready?

On my end looks great, just waiting for a final review cc @ArthurZucker @LysandreJik