Got errors when calling AutoModelForCausalLM related APIs
Author: ztxjack
Pull request: no

### System Info

torch ==1.10.1+cu111
python == 3.8.5
CUDA 11.2
2 GPU == Tesla P40 (vRAM 24G)
14 core CPU == Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
RAM == 112G
ubuntu 18 OS


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I'm now using 4.8.1 transformers codebase for running some LLM models.
While I run into some trouble when trying to switch device to cpu mode.
```
  device = 'cpu'
    model = AutoModelForCausalLM.from_pretrained(
        args.model_id, revision=args.revision, device_map="auto", torch_dtype=torch.float32
    ).to(device)
```
method "to(device)" is what I add to call additionally comparing to the origin code, and for the reason why doing this is that model is by default loaded to the GPU while tokenizer is set to be loaded to CPU.
otherwise showstop message would given as:
```
Loading checkpoint shards: 100%|██████████████████| 4/4 [00:29<00:00,  7.30s/it]
/opt/anaconda3/envs/py385/lib/python3.8/site-packages/transformers/generation/utils.py:1405: 
UserWarning: You are calling .generate() with the `input_ids` being on a device type different than your model's device. 
`input_ids` is on cpu, whereas the model is on cuda. You may experience unexpected behaviors or slower generation. 
Please make sure that you have put `input_ids` to the correct device by calling for example input_ids = input_ids.to('cuda') before running `.generate()`.
```

but after adding "to(device)" and rerun again another problem would show up at model loading.
```
Loading checkpoint shards: 100%|██████████████████| 4/4 [00:29<00:00,  7.32s/it]
Traceback (most recent call last):
  File "generate_cpu.py", line 139, in <module>
    main()
  File "generate_cpu.py", line 113, in main
    model = AutoModelForCausalLM.from_pretrained(
  File "/opt/anaconda3/envs/py385/lib/python3.8/site-packages/transformers/modeling_utils.py", line 1896, in to
    return super().to(*args, **kwargs)
  File "/opt/anaconda3/envs/py385/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1145, in to
    return self._apply(convert)
  File "/opt/anaconda3/envs/py385/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply
    module._apply(fn)
  File "/opt/anaconda3/envs/py385/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply
    module._apply(fn)
  File "/opt/anaconda3/envs/py385/lib/python3.8/site-packages/torch/nn/modules/module.py", line 797, in _apply
    module._apply(fn)
  [Previous line repeated 1 more time]
  File "/opt/anaconda3/envs/py385/lib/python3.8/site-packages/torch/nn/modules/module.py", line 820, in _apply
    param_applied = fn(param)
  File "/opt/anaconda3/envs/py385/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1143, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
NotImplementedError: Cannot copy out of meta tensor; no data!
```

I have no idea why this happened and it is working if just use all GPU for model and tokenizer but I just want to know how it works if using CPU for both since I wish I could leverage the CPU RAM totally which I have only over 40G VRAM in total and it seems to be slow when using GPU only by default.

**Here attaches the target script generate.py:**

```
import argparse

import torch
from dialogues import DialogueTemplate, get_dialogue_template
from transformers import (AutoModelForCausalLM, AutoTokenizer,
                          GenerationConfig, set_seed)


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--model_id",
        type=str,
        help="Name of model to generate samples with",
    )
    parser.add_argument(
        "--revision",
        type=str,
        default=None,
        help="The model repo's revision to use",
    )
    parser.add_argument(
        "--system_prompt", type=str, default=None, help="Overrides the dialogue template's system prompt"
    )
    args = parser.parse_args()

    # Set seed for reproducibility
    set_seed(42)

    prompts = [
        [
            {
                "role": "user",
                "content": "Develop a C++ program that reads a text file line by line and counts the number of occurrences of a specific word in the file.",
            }
        ],
        [
            {
                "role": "user",
                "content": "Implement a Python function to find the longest common subsequence of two input strings using dynamic programming.",
            }
        ],
        [{"role": "user", "content": "Implement a regular expression in Python to validate an email address."}],
        [
            {
                "role": "user",
                "content": "Write a program to find the nth Fibonacci number using dynamic programming.",
            }
        ],
        [
            {
                "role": "user",
                "content": "Implement a binary search algorithm to find a specific element in a sorted array.",
            }
        ],
        [{"role": "user", "content": "Implement a queue data structure using two stacks in Python."}],
        [
            {
                "role": "user",
                "content": "Implement a program to find the common elements in two arrays without using any extra data structures.",
            }
        ],
    ]

    try:
        dialogue_template = DialogueTemplate.from_pretrained(args.model_id, revision=args.revision)
    except Exception:
        print("No dialogue template found in model repo. Defaulting to the `no_system` template.")
        dialogue_template = get_dialogue_template("no_system")

    if args.system_prompt is not None:
        dialogue_template.system = args.system_prompt
    formatted_prompts = []
    for prompt in prompts:
        dialogue_template.messages = [prompt] if isinstance(prompt, dict) else prompt
        formatted_prompts.append(dialogue_template.get_inference_prompt())

    print("=== SAMPLE PROMPT ===")
    print(formatted_prompts[0])
    print("=====================")

    device = "cpu"
    tokenizer = AutoTokenizer.from_pretrained(args.model_id, revision=args.revision)
    print(f"Special tokens: {tokenizer.special_tokens_map}")
    print(f"EOS token ID for generation: {tokenizer.convert_tokens_to_ids(dialogue_template.end_token)}")
    generation_config = GenerationConfig(
        temperature=0.2,
        top_k=50,
        top_p=0.95,
        repetition_penalty=1.2,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.convert_tokens_to_ids(dialogue_template.end_token),
        min_new_tokens=32,
        max_new_tokens=256,
    )
    model = AutoModelForCausalLM.from_pretrained(
        args.model_id, revision=args.revision, device_map="auto", torch_dtype=torch.float32
    ).to(device)
    outputs = ""
    for idx, prompt in enumerate(formatted_prompts):
        batch = tokenizer(prompt, return_tensors="pt", return_token_type_ids=False).to(device)
        generated_ids = model.generate(**batch, generation_config=generation_config)
        generated_text = tokenizer.decode(generated_ids[0], skip_special_tokens=False).lstrip()
        outputs += generated_text + "\n\n"
        print(f"=== EXAMPLE {idx} ===")
        print()
        print(generated_text)
        print()
        print("======================")
        print()



if __name__ == "__main__":
    main()
```


### Expected behavior

1. No error messages is given after doing some code adaption using only CPU for both model and tokenizer.
2. Speed up the model loading and tokenizing process on current code if possible

Comments:
This code cannot work. You can't use `device_map="auto"` and then move back your model to another device.

Hi @sgugger,
Thanks for your comment, so how could i adapt the code for device map setup to suppport CPU loading well in this case, do you have any suggestion?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

![image](https://github.com/huggingface/transformers/assets/145651575/17f8115d-effc-4081-a75f-2e5c47114abe)
![image](https://github.com/huggingface/transformers/assets/145651575/2a1d3687-1c9c-4abc-97ff-9c51ea3c0314)

getting above error with this code of AutoModelForCausalLM , you can check that the accelarate and bitsandbytes already installed , my laptop dont have GPU dooes it create error? guide me to fix this
model_repo = 'daryl149/llama-2-7b-chat-hf'
        
        tokenizer = AutoTokenizer.from_pretrained(model_repo, use_fast=True)

        model = AutoModelForCausalLM.from_pretrained(
            model_repo,
            load_in_4bit=True,
            device_map='auto',
            torch_dtype=torch.float16,
            low_cpu_mem_usage=True,
            trust_remote_code=True
        )
        
        max_len = 2048