[`PEFT`] Peft integration alternative design 
Author: younesbelkada
Pull request: yes

# What does this PR do?

From the offline discussion + the comments from @patrickvonplaten in https://github.com/huggingface/transformers/pull/24827#issuecomment-1641750464 I propose a new design for tightly integrating PEFT into transformers. 
This integration enables loading any PEFT adapter that is saved locally or on the Hub directly into PEFT without dispatching the entire model creation process to PEFT as introduced in #24827. 

This would also enable an easier pipeline integration (a one-liner to load adapter weights) | EDIT: pipeline should work out of the box

Let's constraint this integration to few PEFT methods only, for simplicity and redirect users to use PEFT for advanced features (e.g. merge and unload) and advanced PEFT methods (adaptation prompt, prompt learning).

Current API:

## Load a model with an adapter locally or from the Hub:

```python
import torch
from transformers import AutoModelForCausalLM, OPTForCausalLM

model_id = "facebook/opt-350m"
adapter_model_id = "ybelkada/opt-350m-lora"

# directly on from_pretrained
model = AutoModelForCausalLM.from_pretrained(adapter_model_id)
print(model)

# directly on from_pretrained
model = OPTForCausalLM.from_pretrained(adapter_model_id)
print(model)
```

## Load and attach adapter to an existing model

```python
from transformers import AutoModelForCausalLM

# with load_adapter
model = AutoModelForCausalLM.from_pretrained(model_id)
model.load_adapter(adapter_model_id)

print(model)

# 8-bit + multiGPU compatiblity
model = AutoModelForCausalLM.from_pretrained(model_id, load_in_8bit=True, device_map="balanced")
model.load_adapter(adapter_model_id)

print(model)
print(set(model.hf_device_map.values()))

_ = model(torch.LongTensor([[0, 1, 2, 3]]).to(0))
```

## Attach an adapter, iteratively enable / disable adapters

```python
from transformers import AutoModelForCausalLM, OPTForCausalLM, AutoTokenizer
from peft import PeftConfig

model_id = "facebook/opt-350m"
adapter_model_id = "ybelkada/opt-350m-lora"
tokenizer = AutoTokenizer.from_pretrained(model_id)
text = "Hello"
inputs = tokenizer(text, return_tensors="pt")

model = AutoModelForCausalLM.from_pretrained(model_id)
peft_config = PeftConfig.from_pretrained(adapter_model_id)

# To get random weights
peft_config.init_lora_weights = False

model.add_adapter(peft_config)
print(model)

model.disable_adapters()
output_disabled = model.generate(**inputs)
print(tokenizer.decode(output_disabled[0], skip_special_tokens=True))
>>> Hello, I'm a newbie to this sub. I'm looking for a good place to

model.enable_adapters()
output_enabled = model.generate(**inputs)
print(tokenizer.decode(output_enabled[0], skip_special_tokens=True))
>>> Hello, MMMMMMMM
```

## Add multiple adapters

```python
from transformers import AutoModelForCausalLM, OPTForCausalLM, AutoTokenizer
from peft import PeftConfig, LoraConfig

model_id = "facebook/opt-350m"

# directly on from_pretrained
model = AutoModelForCausalLM.from_pretrained(model_id)

lora_config = LoraConfig(
    target_modules=["q_proj", "k_proj"],
    init_lora_weights=False
)

model.add_adapter(lora_config, adapter_name="adapter_1")

# attach new adapter with same config
model.add_adapter(lora_config, adapter_name="adapter_2")

model.set_adapter("adapter_1")
output_disabled = model.generate(**inputs)
print(tokenizer.decode(output_disabled[0], skip_special_tokens=True))
>>> Hello, I'm a newbie to this sub. I'm looking for a good place to

model.set_adapter("adapter_2")
output_enabled = model.generate(**inputs)
print(tokenizer.decode(output_enabled[0], skip_special_tokens=True))
>>> Hello, I'm a newbie to the game. I'm looking for a good way to
```

## Save adapters

```python
from transformers import AutoModelForCausalLM, OPTForCausalLM, AutoTokenizer
from peft import PeftConfig, LoraConfig

model_id = "facebook/opt-350m"

# directly on from_pretrained
model = AutoModelForCausalLM.from_pretrained(model_id)

lora_config = LoraConfig(
    target_modules=["q_proj", "k_proj"],
)

model.add_adapter(lora_config)

... # train here

model.save_pretrained(save_dir) 

# you can either load it back with transformers or PEFT

from peft import AutoPeftModelForCausalLM

model = AutoPeftModelForCausalLM.from_pretrained(save_dir)

# or

model = AutoModelForCausalLM.from_pretrained(save_dir)
```

## Train adapters using Trainer

Check this gist: https://gist.github.com/younesbelkada/cdda6e4abcb09e58f6324d75e0d88862 

This PR is on par with: https://github.com/huggingface/peft/pull/749

Features to support:

- [x] loading PEFT adapters
- [x] using multiple adapters
- [x] deal with models loaded with accelerate
- [x] Loading directly from `from_pretrained`
- [ ] Merging adapter weights - to not support
- [ ] Unload adapter weights - to not support
- [x] Training with BC with expected PEFT checkpoints format (do we really want to support training? Shall we just redirect users to load a classic `PeftModel` if they want to train a model?)
- [x] What about `save_pretrained` ? 
 
Features to **not** support:

- [x] disabling adapters
- [x] prompt tuning / prompt learning methods

TODOs:

- [ ] docs
- [x] tests

cc @sgugger @patrickvonplaten @BenjaminBossan @pacman100 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Mmmm, but know we can't do `model = AutoModelForCausalLM.from_pretrained("ybelkada/opt-350m-lora")` despite the checkpoint having all the correct info to load the model (compared to the alternative PR). Can we still add the necessary code in `from_pretrained` to have the load take one line instead of two?

Apart from that, the design looks great to me!

At the moment it looks like loading multiple adapters is supported, could we maybe add a couple of examples when multiple adapters are loaded?

@patrickvonplaten 

> At the moment it looks like loading multiple adapters is supported, could we maybe add a couple of examples when multiple adapters are loaded?

Modified the PR description to expose all the possible things that users can do currently

I think that now all core components are available, before adding the tests and the documentation, I would love to have a round of review @patrickvonplaten @sgugger @pacman100 @BenjaminBossan 
I have updated the PR description to detail all possible things users can perform with this integration. Note that there is also a Trainer support and is fully BC with PeftModels

EDIT: I don't know why the CI is currently failing :/ 

Ah, and also this PR should close https://github.com/huggingface/transformers/pull/24750 as pipeline should work out of the box now if you inject adapters or pass a path to an adapter file

Thank you @younesbelkada for the impressive work on adding PEFT as a utility library in Transformers 🔥🚀✨

> # What does this PR do?
> From the offline discussion + the comments from @patrickvonplaten in [#24827 (comment)](https://github.com/huggingface/transformers/pull/24827#issuecomment-1641750464) I propose a new design for tightly integrating PEFT into transformers. This integration enables loading any PEFT adapter that is saved locally or on the Hub directly into PEFT without dispatching the entire model creation process to PEFT as introduced in #24827.
> 
> This would also enable an easier pipeline integration (a one-liner to load adapter weights) | EDIT: pipeline should work out of the box
> 
> Let's constraint this integration to few PEFT methods only, for simplicity and redirect users to use PEFT for advanced features (e.g. merge and unload) and advanced PEFT methods (adaptation prompt, prompt learning).
> 
> Current API:
> 
> ## Load a model with an adapter locally or from the Hub:
> ```python
> import torch
> from transformers import AutoModelForCausalLM, OPTForCausalLM
> 
> model_id = "facebook/opt-350m"
> adapter_model_id = "ybelkada/opt-350m-lora"
> 
> # directly on from_pretrained
> model = AutoModelForCausalLM.from_pretrained(adapter_model_id)
> print(model)
> 
> # directly on from_pretrained
> model = OPTForCausalLM.from_pretrained(adapter_model_id)
> print(model)
> ```
> 
> ## Load and attach adapter to an existing model
> ```python
> from transformers import AutoModelForCausalLM
> 
> # with load_adapter
> model = AutoModelForCausalLM.from_pretrained(model_id)
> model.load_adapter(adapter_model_id)
> 
> print(model)
> 
> # 8-bit + multiGPU compatiblity
> model = AutoModelForCausalLM.from_pretrained(model_id, load_in_8bit=True, device_map="balanced")
> model.load_adapter(adapter_model_id)
> 
> print(model)
> print(set(model.hf_device_map.values()))
> 
> _ = model(torch.LongTensor([[0, 1, 2, 3]]).to(0))
> ```
> 
> ## Attach an adapter, iteratively enable / disable adapters
> ```python
> from transformers import AutoModelForCausalLM, OPTForCausalLM, AutoTokenizer
> from peft import PeftConfig
> 
> model_id = "facebook/opt-350m"
> adapter_model_id = "ybelkada/opt-350m-lora"
> tokenizer = AutoTokenizer.from_pretrained(model_id)
> text = "Hello"
> inputs = tokenizer(text, return_tensors="pt")
> 
> model = AutoModelForCausalLM.from_pretrained(model_id)
> peft_config = PeftConfig.from_pretrained(adapter_model_id)
> 
> # To get random weights
> peft_config.init_lora_weights = False
> 
> model.add_adapter(peft_config)
> print(model)
> 
> model.disable_adapters()
> output_disabled = model.generate(**inputs)
> print(tokenizer.decode(output_disabled[0], skip_special_tokens=True))
> >>> Hello, I'm a newbie to this sub. I'm looking for a good place to
> 
> model.enable_adapters()
> output_enabled = model.generate(**inputs)
> print(tokenizer.decode(output_enabled[0], skip_special_tokens=True))
> >>> Hello, MMMMMMMM
> ```
> 
> ## Add multiple adapters
> ```python
> from transformers import AutoModelForCausalLM, OPTForCausalLM, AutoTokenizer
> from peft import PeftConfig, LoraConfig
> 
> model_id = "facebook/opt-350m"
> 
> # directly on from_pretrained
> model = AutoModelForCausalLM.from_pretrained(model_id)
> 
> lora_config = LoraConfig(
>     target_modules=["q_proj", "k_proj"],
>     init_lora_weights=False
> )
> 
> model.add_adapter(lora_config, adapter_name="adapter_1")
> 
> # attach new adapter with same config
> model.add_adapter(lora_config, adapter_name="adapter_2")
> 
> model.set_adapter("adapter_1")
> output_disabled = model.generate(**inputs)
> print(tokenizer.decode(output_disabled[0], skip_special_tokens=True))
> >>> Hello, I'm a newbie to this sub. I'm looking for a good place to
> 
> model.set_adapter("adapter_2")
> output_enabled = model.generate(**inputs)
> print(tokenizer.decode(output_enabled[0], skip_special_tokens=True))
> >>> Hello, I'm a newbie to the game. I'm looking for a good way to
> ```
> 
> ## Save adapters
> ```python
> from transformers import AutoModelForCausalLM, OPTForCausalLM, AutoTokenizer
> from peft import PeftConfig, LoraConfig
> 
> model_id = "facebook/opt-350m"
> 
> # directly on from_pretrained
> model = AutoModelForCausalLM.from_pretrained(model_id)
> 
> lora_config = LoraConfig(
>     target_modules=["q_proj", "k_proj"],
> )
> 
> model.add_adapter(lora_config)
> 
> ... # train here
> 
> model.save_pretrained(save_dir) 
> 
> # you can either load it back with transformers or PEFT
> 
> from peft import AutoPeftModelForCausalLM
> 
> model = AutoPeftModelForCausalLM.from_pretrained(save_dir)
> 
> # or
> 
> model = AutoModelForCausalLM.from_pretrained(save_dir)
> ```
> 
> ## Train adapters using Trainer
> Check this gist: https://gist.github.com/younesbelkada/cdda6e4abcb09e58f6324d75e0d88862
> 
> This PR is on par with: [huggingface/peft#749](https://github.com/huggingface/peft/pull/749)
> 
> Features to support:
> 
> * [x]  loading PEFT adapters
> * [x]  using multiple adapters
> * [x]  deal with models loaded with accelerate
> * [x]  Loading directly from `from_pretrained`
> * [ ]  Merging adapter weights - to not support
> * [ ]  Unload adapter weights - to not support
> * [x]  Training with BC with expected PEFT checkpoints format (do we really want to support training? Shall we just redirect users to load a classic `PeftModel` if they want to train a model?)
> * [x]  What about `save_pretrained` ?
> 
> Features to **not** support:
> 
> * [x]  disabling adapters
> * [x]  prompt tuning / prompt learning methods
> 
> TODOs:
> 
> * [ ] docs
> * [x]  tests
> 
> cc @sgugger @patrickvonplaten @BenjaminBossan @pacman100

What packages to install do i need to run this code?

> What packages to install do i need to run this code?

Just install the latest `transformers` & `peft`

```bash
pip install -U peft transformers
```