trainer.is_model_parallel seems conflict with deepspeed
Author: ehion
Pull request: no

### System Info

accelerate                    0.18.0
aiofiles                      23.1.0
aiohttp                       3.8.4
aiosignal                     1.3.1
altair                        4.2.2
anyio                         3.6.2
asttokens                     2.2.1
async-timeout                 4.0.2
attrs                         22.2.0
backcall                      0.2.0
backports.functools-lru-cache 1.6.4
bcrypt                        4.0.1
bitsandbytes                  0.37.2
certifi                       2022.12.7
cfgv                          3.3.1
chardet                       5.1.0
charset-normalizer            3.0.1
click                         8.1.3
colossalai                    0.2.5
comm                          0.1.3
contexttimer                  0.3.3
contourpy                     1.0.7
cPython                       0.0.6
cycler                        0.11.0
datasets                      2.11.0
debugpy                       1.6.7
decorator                     5.1.1
deepspeed                     0.9.0
dill                          0.3.6
distlib                       0.3.6
dnspython                     2.3.0
entrypoints                   0.4
evaluate                      0.4.0
executing                     1.2.0
fabric                        3.0.0
fastapi                       0.95.0
ffmpy                         0.3.0
filelock                      3.11.0
fonttools                     4.39.3
frozenlist                    1.3.3
fsspec                        2023.4.0
gradio                        3.24.1
gradio_client                 0.0.8
h11                           0.14.0
hjson                         3.1.0
httpcore                      0.16.3
httpx                         0.23.3
huggingface-hub               0.13.4
identify                      2.5.18
idna                          3.4
importlib-metadata            6.3.0
importlib-resources           5.12.0
invoke                        2.0.0
ipykernel                     6.22.0
ipython                       8.12.0
jedi                          0.18.2
Jinja2                        3.1.2
joblib                        1.2.0
jsonschema                    4.17.3
jupyter_client                8.1.0
jupyter_core                  5.3.0
kiwisolver                    1.4.4
linkify-it-py                 2.0.0
loralib                       0.1.1
markdown-it-py                2.2.0
MarkupSafe                    2.1.2
matplotlib                    3.7.1
matplotlib-inline             0.1.6
mdit-py-plugins               0.3.3
mdurl                         0.1.2
mpi4py                        3.1.4
multidict                     6.0.4
multiprocess                  0.70.14
nest-asyncio                  1.5.6
ninja                         1.11.1
nodeenv                       1.7.0
numpy                         1.24.2
nvidia-cublas-cu11            11.10.3.66
nvidia-cuda-nvrtc-cu11        11.7.99
nvidia-cuda-runtime-cu11      11.7.99
nvidia-cudnn-cu11             8.5.0.96
orjson                        3.8.10
packaging                     23.1
pandas                        2.0.0
paramiko                      3.0.0
parso                         0.8.3
peft                          0.2.0
pexpect                       4.8.0
pickleshare                   0.7.5
Pillow                        9.5.0
pip                           23.0.1
platformdirs                  3.2.0
pre-commit                    3.1.0
prompt-toolkit                3.0.38
psutil                        5.9.4
ptyprocess                    0.7.0
pure-eval                     0.2.2
py-cpuinfo                    9.0.0
pyarrow                       10.0.0
pydantic                      1.10.7
pydub                         0.25.1
Pygments                      2.15.0
pymongo                       4.3.3
PyNaCl                        1.5.0
pyparsing                     3.0.9
pyrsistent                    0.19.3
python-dateutil               2.8.2
python-multipart              0.0.6
pytz                          2023.3
PyYAML                        6.0
pyzmq                         25.0.2
regex                         2022.10.31
requests                      2.28.2
responses                     0.18.0
rfc3986                       1.5.0
rich                          13.3.1
scikit-learn                  1.2.2
scipy                         1.10.1
semantic-version              2.10.0
sentencepiece                 0.1.97
setuptools                    67.6.0
six                           1.16.0
sniffio                       1.3.0
stack-data                    0.6.2
starlette                     0.26.1
threadpoolctl                 3.1.0
tokenizers                    0.13.2
toolz                         0.12.0
torch                         1.13.1
tornado                       6.2
tqdm                          4.65.0
traitlets                     5.9.0
transformers                  4.28.0.dev0
typing_extensions             4.5.0
tzdata                        2023.3
uc-micro-py                   1.0.1
urllib3                       1.26.15
uvicorn                       0.21.1
virtualenv                    20.19.0
wcwidth                       0.2.6
websockets                    11.0.1
wheel                         0.40.0
xxhash                        3.2.0
yarl                          1.8.2
zipp                          3.15.0

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

python -m torch.distributed.launch $DISTRIBUTED_ARGS run_clm.py --model_name_or_path "/mnt/zts-dev-data/llama-7b-hf/" --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 --per_device_train_batch_size 1 --per_device_eval_batch_size 1 --do_train --do_eval --output_dir /mnt/zts-dev-data/tmp/test-clm --tokenizer_name test --logging_steps 50 --save_steps 1000 --overwrite_output_dir --fp16 True --deepspeed deepspeed_test.json 

### Expected behavior

It's ok for me when i pretrained a llama 7b model with 2*a100 with no deepspeed(OOM when only 1*a100 training and huggingface would support model parallel by default therefore no OOM occured), but when i configured training script with --deepspeed this error appeared:

```python
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument index in method wrapper__index_select)
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 70758 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 70757) of binary: 
``` 

```
{
    "fp16": {
        "enabled": true,
        "loss_scale": 0,
        "loss_scale_window": 1000,
        "initial_scale_power": 16,
        "hysteresis": 2,
        "min_loss_scale": 1
    },
    "optimizer": {
        "type": "AdamW",
        "params": {
            "lr": "auto",
            "weight_decay": "auto",
            "torch_adam": true,
            "adam_w_mode": true
        }
    },
    "scheduler": {
        "type": "WarmupDecayLR",
        "params": {
            "warmup_min_lr": "auto",
            "warmup_max_lr": "auto",
            "warmup_num_steps": "auto",
            "total_num_steps": "auto"
        }
    },
    "zero_optimization": {
        "stage": 1,
        "offload_optimizer": {
         "device": "cpu",
         "pin_memory": true
        },
        "allgather_partitions": true,
        "allgather_bucket_size": 2e8,
        "overlap_comm": true,
        "reduce_scatter": true,
        "reduce_bucket_size": "auto",
        "contiguous_gradients": true
    },
    "gradient_accumulation_steps": "auto",
    "gradient_clipping": "auto",
    "steps_per_print": 2000,
    "train_batch_size": "auto",
    "train_micro_batch_size_per_gpu": "auto",
    "wall_clock_breakdown": false
}

```
zero_optimization with stage 2 also errors

Comments:
https://huggingface.co/transformers/v4.6.0/_modules/transformers/trainer.html

and when i test a smaller  model which could fit in one gpu memory and does not need model parallel 
by running :
deepspeed "script + config" --deepspeed **.json (with num_gpus 2 )
no error occures anymore

cc @stas00 

Please provide a reproducible example that I can run to see what the problem is and I would be happy to look at it - I can't do it with what you shared. Thank you.

@stas00  this problem ocures when i instance model with parameter device_map="auto" and no bug when i do not use device_map，device_map will use model parallel when model size is too big for one gpu to hold  in a multigpu-training env。
You can test it just use run_clm.py in transformers and use a large model(one gpu memory could not save the model eg:llama7b)instanced with parameter device_map="auto"
```
model = LlamaForCausalLM.from_pretrained(model_args.model_name_or_path,torch_dtype=torch.float32,
        device_map="auto",
        load_in_8bit=False)
``` 
so previous bug may be a conflict between deepspeed and transformers，transformers build model parallel by default in my setting but deepspeed backend algorithms don't when device_map="auto"

> @stas00 this problem ocures when i instance model with parameter device_map="auto" and no bug when i do not use device_map，device_map will use model parallel when model size is too big for one gpu to hold in a multigpu-training env。 You can test it just use run_clm.py in transformers and use a large model(one gpu memory could not save the model eg:llama7b)instanced with parameter device_map="auto"
> 
> ```
> model = LlamaForCausalLM.from_pretrained(model_args.model_name_or_path,torch_dtype=torch.float32,
>         device_map="auto",
>         load_in_8bit=False)
> ```
> 
> so previous bug may be a conflict between deepspeed and transformers，transformers build model parallel by default in my setting but deepspeed backend algorithms don't when device_map="auto"

Hey, the same situation happend to me, did you solve the problem?

`device_map="auto"` and DeepSpeed are incompatible. You cannot use them together.