`MPTForCausalLM` does not support `device_map='auto'` yet. 
Author: harikc456
Pull request: no

### System Info

- `transformers` version: 4.30.0.dev0
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.6.9 (gpu)
- Jax version: 0.4.10
- JaxLib version: 0.4.10
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@ArthurZucker @younesbelkada 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
import torch
from transformers import AutoModelForCausalLM, AutoConfig
from transformers import BitsAndBytesConfig

model_name = 'mosaicml/mpt-7b-instruct'

nf4_config = BitsAndBytesConfig(
   load_in_4bit=True,
   bnb_4bit_quant_type="nf4",
   bnb_4bit_use_double_quant=True,
   bnb_4bit_compute_dtype=torch.bfloat16
)

model_nf4 = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True, offload_folder="offload", 
                                                 offload_state_dict = True, quantization_config=nf4_config)
```
Error
```
ValueError: MPTForCausalLM does not support `device_map='auto'` yet.
```
I saw a similar issue #22188 for `XGLMForCausalLM`. I couldn't find `MPTForCausalLM ` in the repository if the MPT model is not supported currently, is there any hack that I can use to get `acclerate` support ? Than

### Expected behavior

Model gets loaded without any errors

Comments:
Hey! This is a duplicate of #23784 . Seems like we should give a more informative message. 

This makes me think that we should add the ability to pass no split modules directly when calling from_pretrained for super users. There are more and more models that uses code on the Hub feature and this should make life much easier for these users (sometimes it takes a lot of time for the authors to approve / merge these PRs) wdyt @ArthurZucker @sgugger ?

Hi @harikc456 

You can check: https://github.com/huggingface/transformers/pull/23896#issuecomment-1570036714 

To illustrate what should be done, [I made a PR on the Hub directly,](https://huggingface.co/mosaicml/mpt-7b/discussions/45) you can load the mpt-7b model as follows (until the authors will merge my PR):

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = 'mosaicml/mpt-7b'

tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neox-20b")
model = AutoModelForCausalLM.from_pretrained(
    model_name, 
    load_in_8bit=True,
    device_map="auto",
    trust_remote_code=True,
    revision="pr/45"
)

prompt = "What is the boiling point of Nitrogen?"

input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(0)
out = model.generate(input_ids)
print(tokenizer.decode(out[0], skip_special_tokens=True))
```

> you can load the mpt-7b model as follows (until the authors will merge my PR):

To clarify, the code will continue working after the PR is merged, but you will also be able to do the same thing without `revision="pr/45"`.