load_adapter method device setting bug in the from_pretrained
Author: ljm565
Pull request: no
Labels: name: Core: Modeling

name: bug

name: PEFT

### System Info

python==3.11.9
transformers==4.39.3
peft==0.12.0

### Who can help?

_No response_

### Information
- The official transformers and peft libraries.

### Tasks
- from_pretrained method in the PreTrainedModel class.

### Reproduction

Problem:
I am trying to load a model onto the second GPU (index 2) of an 8-GPU server. I have already set the device using `torch.cuda.set_device(2)`. However, when I load the adapter using `from_pretrained`, the adapter is loaded onto GPU 0, while the model is correctly loaded onto GPU 2.
![Screenshot from 2024-09-27 13-28-20](https://github.com/user-attachments/assets/fa74dff4-6d2a-48e2-8c45-4a4c9a6ab7d0)



Bug Occurrence:
Even though I am specifying the `device_map` as `cuda:2` in `from_pretrained`, when an adapter_model_path is provided, the following part of the code does not pass the `device_map`, and internally, it defaults to `auto`. As a result, the adapter gets loaded onto GPU 0 after passing through the function below.
![Screenshot from 2024-09-27 13-24-44](https://github.com/user-attachments/assets/cee63160-e05d-40da-a8f2-5bea1190bd6a)

More specifically, the `load_peft_weights` does not give specific device_map in the `peft.py` line 193.
![Screenshot from 2024-09-27 13-48-26](https://github.com/user-attachments/assets/f08c67be-9a3c-4b0b-b628-667663535286)


### Expected behavior

It must be loaded on the device specified in `from_pretrained`.


Comments:
cc @BenjaminBossan and @SunMarc 

Thanks for reporting this @ljm565. Could you please provide code to reproduce this behavior?

@BenjaminBossan 
Sorry for the late reply.
Here is the code.

```python
import torch

from transformers import LlamaModel     # transformers==4.39.3
from peft import PeftModel      # peft==0.12.0



# Device environment setup
device_id = 2
device = torch.device(f'cuda:{device_id}')
torch.cuda.set_device(device_id)

# Load model (abnormal case)
# If the pretrained model includes peftconfig, the peft adapter is loaded on the 0th GPU. 
model = LlamaModel.from_pretrained(
            "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp",
            device_map=device,
            load_in_8bit=True,
            low_cpu_mem_usage=True,
            use_cache=True,
        )


# Load model (normal case)
model = LlamaModel.from_pretrained(
            "meta-llama/Meta-Llama-3-8B-Instruct",
            device_map=device,
            load_in_8bit=True,
            low_cpu_mem_usage=True,
            use_cache=True,
        )
```

For "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp" the Peft adapter is loaded on 0th GPU even though a 2nd GPU is used.

For  "meta-llama/Meta-Llama-3-8B-Instruct", it works normally because it does not contain Peft adapter.

Thanks.



Thanks for providing this code. I could not reproduce the error (on a machine with 2 GPUs). Regardless of `device_id` I chose, the adapter was always loaded on the same device as the base model.

I think the issue is that your transformers version is too old and does not contain this fix: https://github.com/huggingface/transformers/commit/367a0dbd53cc1b826d986b166f3ac520d500db64. If you upgrade transformers, the error should vanish.

Thanks for your reply. I upgraded the version as you said and it works.