auto move input to device of the first-layer if necessary
Author: ranchlai
Pull request: yes

# What does this PR do?
This PR adds feature to model.generate()  to check if input_ids is at the same device of input_embeddings. 
If not, move input to the device rather than simply raising an error or warning.
It should be helpful when device_map="auto" or when there are multi-gpus to choose from, in which case we only need to worry about where to place to model. 

Hope it is acceptable. if not, I will close it.  




Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi @ranchlai, thanks to opening this PR! 

Could you share a code snippet that we can run which would currently fail on main and which runs with this update? 

cc @gante 


@amyeroberts We don't automatically move tensors in simple forward passes, with or without ` device_map="auto"`, so I don't see why we should do it in `generate` 🤗 

(And, if we do decide do move the tensors in the forward pass, `generate` would automatically benefit from it :D)

sure. Before the PR, 
```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "mosaicml/mpt-7b-chat"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name, 
    llm_int8_enable_fp32_cpu_offload=True,
    load_in_8bit=True,
    device_map="auto")

text = "Write a python program to find the largest prime number below 1000."
input_ids = tokenizer.encode(text, return_tensors="pt")
output = model.generate(input_ids, max_length=100, do_sample=True)
response = tokenizer.decode(output[0])
print(response)
```
Error trace: 
```
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:05<00:00,  2.84s/it]
[INFO|/home/ranch/models/transformers/src/transformers/modeling_utils.py:3383] 2023-07-31 23:53:40,770 >> All model checkpoint weights were used when initializing MptForCausalLM.

[INFO|/home/ranch/models/transformers/src/transformers/modeling_utils.py:3391] 2023-07-31 23:53:40,770 >> All the weights of MptForCausalLM were initialized from the model checkpoint at mosaicml/mpt-7b-chat.
If your task is similar to the task the model of the checkpoint was trained on, you can already use MptForCausalLM for predictions without further training.
[INFO|/home/ranch/models/transformers/src/transformers/generation/configuration_utils.py:576] 2023-07-31 23:53:40,771 >> loading configuration file mosaicml/mpt-7b-chat/generation_config.json
[INFO|/home/ranch/models/transformers/src/transformers/generation/configuration_utils.py:616] 2023-07-31 23:53:40,771 >> Generate config GenerationConfig {
  "_from_model_config": true,
  "eos_token_id": [
    0,
    50278
  ],
  "transformers_version": "4.32.0.dev0",
  "use_cache": false
}

[INFO|/home/ranch/models/transformers/src/transformers/generation/configuration_utils.py:616] 2023-07-31 23:53:40,786 >> Generate config GenerationConfig {
  "_from_model_config": true,
  "transformers_version": "4.32.0.dev0",
  "use_cache": false
}

/home/ranch/models/transformers/src/transformers/generation/utils.py:1296: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation )
  warnings.warn(
/home/ranch/models/transformers/src/transformers/generation/utils.py:1501: UserWarning: You are calling .generate() with the `input_ids` being on a device type different than your model's device. `input_ids` is on cpu, whereas the model is on cuda. You may experience unexpected behaviors or slower generation. Please make sure that you have put `input_ids` to the correct device by calling for example input_ids = input_ids.to('cuda') before running `.generate()`.
  warnings.warn(
Traceback (most recent call last):
  File "/home/ranch/models/auto_move_input_ids_to_devcie/mpt.py", line 13, in <module>
    output = model.generate(input_ids, max_length=100, do_sample=True)
  File "/media/ranch/sda1/anaconda3/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/ranch/models/transformers/src/transformers/generation/utils.py", line 1622, in generate
    return self.sample(
  File "/home/ranch/models/transformers/src/transformers/generation/utils.py", line 2765, in sample
    next_token_scores = logits_warper(input_ids, next_token_scores)
  File "/home/ranch/models/transformers/src/transformers/generation/logits_process.py", line 97, in __call__
    scores = processor(input_ids, scores)
  File "/home/ranch/models/transformers/src/transformers/generation/logits_process.py", line 388, in __call__
    indices_to_remove = scores < torch.topk(scores, top_k)[0][..., -1, None]
RuntimeError: "topk_cpu" not implemented for 'Half'
```
After PR: 
```
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:06<00:00,  3.01s/it]
[INFO|/home/ranch/models/transformers/src/transformers/modeling_utils.py:3383] 2023-07-31 23:55:37,080 >> All model checkpoint weights were used when initializing MptForCausalLM.

[INFO|/home/ranch/models/transformers/src/transformers/modeling_utils.py:3391] 2023-07-31 23:55:37,080 >> All the weights of MptForCausalLM were initialized from the model checkpoint at mosaicml/mpt-7b-chat.
If your task is similar to the task the model of the checkpoint was trained on, you can already use MptForCausalLM for predictions without further training.
[INFO|/home/ranch/models/transformers/src/transformers/generation/configuration_utils.py:576] 2023-07-31 23:55:37,081 >> loading configuration file mosaicml/mpt-7b-chat/generation_config.json
[INFO|/home/ranch/models/transformers/src/transformers/generation/configuration_utils.py:616] 2023-07-31 23:55:37,081 >> Generate config GenerationConfig {
  "_from_model_config": true,
  "eos_token_id": [
    0,
    50278
  ],
  "transformers_version": "4.32.0.dev0",
  "use_cache": false
}

[INFO|/home/ranch/models/transformers/src/transformers/generation/configuration_utils.py:616] 2023-07-31 23:55:37,097 >> Generate config GenerationConfig {
  "_from_model_config": true,
  "transformers_version": "4.32.0.dev0",
  "use_cache": false
}

/home/ranch/models/transformers/src/transformers/generation/utils.py:1296: UserWarning: You have modified the pretrained model configuration to control generation. This is a deprecated strategy to control generation and will be removed soon, in a future version. Please use a generation configuration file (see https://huggingface.co/docs/transformers/main_classes/text_generation )
  warnings.warn(
[INFO|/home/ranch/models/transformers/src/transformers/generation/utils.py:1339] 2023-07-31 23:55:37,097 >> Moving input tensor from device `cpu` to `cuda:0`
Write a python program to find the largest prime number below 1000.
We are going to learn how to create a program in python to find largest prime number in O(logN) time using Sieve of Eratosthenes. Sieve of Eratosthenes is an efficient algorithm for making the list of prime numbers. We will learn how to create a python program to find the largest prime number (sieve of eratosthenes).
Eratosthenes is
````

> @amyeroberts We don't automatically move tensors in simple forward passes, with or without ` device_map="auto"`, so I don't see why we should do it in `generate` 🤗
> 
> (And, if we do decide do move the tensors in the forward pass, `generate` would automatically benefit from it :D)

Or think it another way? please let me know if I am not thinking clearly. ^_^
Advanced function/class such as  `.generate()` or `Pipeline()`, receive more information such as user inputs / device informations than `.forward()`. Hence, these functions/classes should know better on how to make use of resources.

If we know that we can safely(which I also need comments on whether it is safe to move or not), why not just move instead of raising an error? 

@ranchlai our `transformers` [philosophy](https://huggingface.co/docs/transformers/philosophy) dictates that lower-level interfaces like `.forward()` or `.generate()` avoid hidden/implicit transformations (like setting the right device), but higher-level interfaces like the `pipeline()` may do it :)

Thank you for comments @gante I thought generate is high-level.  Will close. 

@ranchlai no worries, new proposals are always welcome 🤗