Llama3 with LlamaForSequenceClassification - Shape mismatch error
Author: parasurama
Pull request: no

### System Info

- `transformers` version: 4.40.0
- Platform: Linux-5.14.0-284.40.1.el9_2.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.11
- Huggingface_hub version: 0.22.2
- Safetensors version: 0.4.2
- Accelerate version: 0.29.3
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.1+cu121 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

### Who can help?

@ArthurZucker  @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I'm getting a shape mismatch error when loading the LLama3 model with LlamaForSequenceClassification


```
model_path = "meta-llama/Meta-Llama-3-8B"
    
  max_length = 4096
  quant_config = BitsAndBytesConfig(
      load_in_4bit=True,
      bnb_4bit_quant_type="nf4",
      bnb_4bit_compute_dtype=torch.bfloat16,
      bnb_4bit_use_double_quant=False,
  )

  peft_config = LoraConfig(task_type=TaskType.SEQ_CLS, 
                           inference_mode=False, 
                           lora_alpha=LORA_ALPHA,
                           lora_dropout=LORA_DROPOUT,
                           r=LORA_R,
                           bias="none",
                           target_modules=LORA_TARGET_MODULES)
  
 llama_config = LlamaConfig(max_position_embeddings=max_length)

model = LlamaForSequenceClassification.from_pretrained(model_path,
                                                       config=llama_config,
                                                       quantization_config=quant_config,
                                                       ignore_mismatched_sizes=True)
```

### Expected behavior

Returns the following error

```
Some weights of LlamaForSequenceClassification were not initialized from the model checkpoint at meta-llama/Meta-Llama-3-8B and are newly initialized because the shapes did not match
```

Comments:
Hey! Pretty sure that is expected: there are no pretrained checkpoints on sequence classification no? 
THis does look like a warning not an error

You're right -- this was a warning not an error. 

The warning is followed by the following error.

```
ValueError: weight is on the meta device, we need a `value` to put in on 0.
```

It looks like there are similar open issues. Something to do with tied weights not being detected correctly

https://github.com/huggingface/accelerate/issues/2059


cc @SunMarc 

Hi @parasurama, thanks for reporting ! I'll have a look asap


Hi @parasurama, this happens because you changed `max_position_embeddings` attribute. This modified a lot of weights and the whole model needs to be retrained. For now, we don't support loading mismatched weights with `device_map="auto"`. I will work on adding that feature but this is not very useful since `device_map` should be used for inference mainly and the modified model needs retraining. 

@SunMarc the error is raised even if I use the default `max_position_embeddings`.
The error is not raised when using a LLama2 model.

Here's a minimum example:

```
from transformers import (
    LlamaForSequenceClassification,
    LlamaConfig,
    BitsAndBytesConfig,
)
from peft import LoraConfig, TaskType
import torch
    
quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=False,
)

peft_config = LoraConfig(task_type=TaskType.SEQ_CLS, 
                            inference_mode=False, 
                            lora_alpha=16,
                            lora_dropout=0.05,
                            r=16,
                            bias="none",
                            target_modules=["q_proj", "v_proj"])
    
llama_config = LlamaConfig()

model = LlamaForSequenceClassification.from_pretrained("meta-llama/Meta-Llama-3-8B",
                                                        config=llama_config,
                                                        quantization_config=quant_config,
                                                        ignore_mismatched_sizes=True,
                                                        )
```

This happens because the default vocab_size of LlamaConfig is 32000 but llama v3 checkpoint have a vocab_size of 128256 but llama v2 checkpoint have a vocab_size of 32000. So by passing LlamaConfig() with "meta-llama/Meta-Llama-3-8B", you are modifying the model. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @parasurama, is the issue solved ? LMK if I can chose the issue ! 

Hi @SunMarc 
Yes, I had to load the llama3 config file from source. That resolved the issue. 
Thanks!