Loading LLM LoRA locally does not update weights 
Author: JohnnyRacer
Pull request: no

### System Info

- `transformers` version: 4.28.1
- Platform: Linux-5.4.0-148-generic-x86_64-with-glibc2.17
- Python version: 3.8.16
- Huggingface_hub version: 0.12.0
- Safetensors version: not installed
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?:  no

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

After LoRA training and saving the model with the following snippet:
```py
import torch
from transformers import LlamaTokenizer, LlamaForCausalLM,GenerationConfig

tokenizer = LlamaTokenizer.from_pretrained('decapoda-research/llama-7b-hf')
model = LlamaForCausalLM.from_pretrained('decapoda-research/llama-7b-hf', device_map="auto", torch_dtype=torch.float16)

from peft import LoraConfig, get_peft_model,TaskType
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)
model = get_peft_model(model, lora_config)

# After trainer.train() is complete

model.save_pretrained('./lora_pretrained')
```

Loading LoRA from save directory:
```py
# Loading base model is same as the snippet above

lora_config = LoraConfig.from_pretrained('./lora_pretrained')
model = get_peft_model(model, lora_config)
#The model generates outputs that are the same as the base model.
```

Trying to load the `adapter_model.bin` directly via this snippet results in errors about incompatible weights:
```py
model.load_state_dict(torch.load('./lora_pretrained/adapter_model.bin'))
RuntimeError: Error(s) in loading state_dict for PeftModelForCausalLM:
	Missing key(s) in state_dict: "base_model.model.model.embed_tokens.weight", "base_model.model.model.layers.0.self_attn.q_proj.weight", "base_model.model.model.layers.0.self_attn.q_proj.lora_A.weight", "base_model.model.model.layers.0.self_attn.q_proj.lora_B.weight", "base_model.model.model.layers.0.self_attn.k_proj.weight", "base_model.model.model.layers.0.self_attn.v_proj.weight", "base_model.model.model.layers.0.self_attn.v_proj.lora_A.weight", "base_model.model.model.layers.0.self_attn.v_proj.lora_B.weight", "base_model.model.model.layers.0.self_attn.o_proj.weight", "base_model.model.model.layers.0.self_attn.rotary_emb.inv_freq", "base_model.model.model.layers.0.mlp.gate_proj.weight", "base_model.model.model.layers.0.mlp.down_proj.weight", "base_model.model.model.layers.0.mlp.up_proj.weight", "base_model.model.model.layers.0.input_layernorm.weight",
```

### Expected behavior

`LoraConfig.from_pretrained` should load the updated model weights.

Comments:
cc @pacman100 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

you need to load both the base model weights `pytorch_model.bin` and the adapter model weights `adapter_model.bin`.

```
from peft import PeftModel
model = PeftModel.from_pretrained(model, save_path) #save_path to contain both adapter_config.json and adapter_model.bin
```

this worked for me