RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cpu
Author: damugongzai
Pull request: no

### System Info

# System Info
python: 3.10
transformers: 4.32.0.dev0
deepspeed: 0.10.0
GPUs: 8 x titanxp

# Reproduction
running the demo to use the deepspeed MOQ algorithm
`python text-classification/run_glue.py   --model_name_or_path bert-base-cased   --task_name $TSK   --do_train   --do_eval   --max_seq_length 128   --per_device_train_batch_size 32   --learning_rate 2e-5   --num_train_epochs 3   --output_dir ./tmp/$TSK/   --fp16   --warmup_steps 2   --deepspeed test.json`
meet the bug
```
Traceback (most recent call last):
  File "/mnt/cephfs/home/zhengzekang/python/project/transformers/examples/pytorch/text-classification/run_glue.py", line 648, in <module>
    main()
  File "/mnt/cephfs/home/zhengzekang/python/project/transformers/examples/pytorch/text-classification/run_glue.py", line 556, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/mnt/cephfs/home/zhengzekang/anaconda3/envs/deepspeed/lib/python3.10/site-packages/transformers/trainer.py", line 1546, in train
    return inner_training_loop(
  File "/mnt/cephfs/home/zhengzekang/anaconda3/envs/deepspeed/lib/python3.10/site-packages/transformers/trainer.py", line 1830, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/mnt/cephfs/home/zhengzekang/anaconda3/envs/deepspeed/lib/python3.10/site-packages/transformers/trainer.py", line 2676, in training_step
    loss = self.compute_loss(model, inputs)
  File "/mnt/cephfs/home/zhengzekang/anaconda3/envs/deepspeed/lib/python3.10/site-packages/transformers/trainer.py", line 2701, in compute_loss
    outputs = model(**inputs)
  File "/mnt/cephfs/home/zhengzekang/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/mnt/cephfs/home/zhengzekang/anaconda3/envs/deepspeed/lib/python3.10/site-packages/torch/nn/parallel/data_parallel.py", line 154, in forward
    raise RuntimeError("module must have its parameters and buffers "
RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cpu
```

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

running the `python text-classification/run_glue.py   --model_name_or_path bert-base-cased   --task_name $TSK   --do_train   --do_eval   --max_seq_length 128   --per_device_train_batch_size 32   --learning_rate 2e-5   --num_train_epochs 3   --output_dir ./tmp/$TSK/   --fp16   --warmup_steps 2   --deepspeed test.json`

### Expected behavior

 thank you for your helping

Comments:
Can you show us the content of `test.json`? Thanks
cc @pacman100 

Hello, please use distributed launcher like `torchrun` or `deepspeed` or `accelerate launch`. Above it is trying to run data parallel with DeepSpeed config which is incorrect.

Here is the test.json
```
{

    "gradient_clipping": 1.0,
    "fp16": {
      "initial_scale_power": 16,
      "enabled": true
    },
    "quantize_training": {
      "enabled": true,
      "quantize_verbose": true,
      "quantizer_kernel": true,
      "quantize_type": "symmetric",
      "quantize_bits": {
        "start_bits": 16,
        "target_bits": 8
      },
      "quantize_schedule": {
        "quantize_period": 10,
        "schedule_offset": 0
      },
      "quantize_groups": 8,
      "fp16_mixed_quantize": {
        "enabled": false,
        "quantize_change_ratio": 0.001
      },
      "eigenvalue": {
        "enabled": true,
        "verbose": true,
        "max_iter": 50,
        "tol": 1e-2,
        "stability": 0,
        "gas_boundary_resolution": 1,
        "layer_name": "bert.encoder.layer",
        "layer_num": 12
      }
    },
    "zero_optimization": {
      "stage": 2,
      "offload_optimizer": {
          "device": "cpu",
          "pin_memory": true
      },
      "allgather_partitions": true,
      "allgather_bucket_size": 2e8,
      "overlap_comm": true,
      "reduce_scatter": true,
      "reduce_bucket_size": 2e8,
      "contiguous_gradients": true
  }
}

```

@pacman100 Thanks. But this is the official demo code given by the deepspeed, can you give a correct command to use `torchrun `or `deepspeed` or `accelerate` launch

Here is the official code link referenced.
<https://www.deepspeed.ai/tutorials/MoQ-tutorial/>

Please refer https://huggingface.co/docs/transformers/main_classes/deepspeed#deployment-with-multiple-gpus

Alternatively select a single GPU via `CUDA_VISIBLE_DEVICES=0` for working with the code link you shared

@pacman100 
Thank you for your patience! Now I successfully running  the demo with
 `deepspeed --num_gpus=2 text-classification/run_glue.py   --model_name_or_path bert-base-cased   --task_name $TSK   --do_train   --do_eval   --max_seq_length 128   --per_device_train_batch_size 32   --learning_rate 2e-5   --num_train_epochs 3   --output_dir ./tmp/$TSK/   --fp16   --warmup_steps 2   --deepspeed test.json --overwrite_output_dir`
But I find the output result and model size is not as expected. No matter how I change the config json, even setring the target bit to -1 as follows, the program still runs successfully.
```
{
    "train_micro_batch_size_per_gpu":"auto",
    "gradient_clipping": 1.0,
    "fp16": {
      "initial_scale_power": 16,
      "enabled": true
    },
    "quantize_training": {
      "enabled": true,
      "quantize_verbose": true,
      "quantizer_kernel": true,
      "quantize-algo": {
        "q_type": "symmetric"
      },
      "quantize_bits": {
        "start_bits": 16,
        "target_bits": -1
      },
      "quantize_schedule": {
        "quantize_period": 400,
        "schedule_offset": 0
      },
      "quantize_groups": 8
    },
    "zero_optimization": {
      "stage": 0
  },
    "scheduler": {
      "type": "WarmupLR",
      "params": {
          "warmup_min_lr": "auto",
          "warmup_max_lr": "auto",
          "warmup_num_steps": "auto"
      }
  }
}
```
here is the running result
```
***** train metrics *****
  epoch                    =        3.0
  train_loss               =     0.4933
  train_runtime            = 0:01:19.65
  train_samples            =       3668
  train_samples_per_second =     138.14
  train_steps_per_second   =      2.184
08/18/2023 15:37:19 - INFO - __main__ - *** Evaluate ***
[INFO|trainer.py:752] 2023-08-18 15:37:19,465 >> The following columns in the evaluation set don't have a corresponding argument in `BertForSequenceClassification.forward` and have been ignored: sentence2, sentence1, idx. If sentence2, sentence1, idx are not expected by `BertForSequenceClassification.forward`,  you can safely ignore this message.
[INFO|trainer.py:3110] 2023-08-18 15:37:19,471 >> ***** Running Evaluation *****
[INFO|trainer.py:3112] 2023-08-18 15:37:19,471 >>   Num examples = 408
[INFO|trainer.py:3115] 2023-08-18 15:37:19,471 >>   Batch size = 8
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 26/26 [00:00<00:00, 26.59it/s]
***** eval metrics *****
  epoch                   =        3.0
  eval_accuracy           =     0.8358
  eval_combined_score     =     0.8604
  eval_f1                 =     0.8851
  eval_loss               =     0.3889
  eval_runtime            = 0:00:01.02
  eval_samples            =        408
  eval_samples_per_second =     399.22
  eval_steps_per_second   =      25.44
[2023-08-18 15:37:21,428] [INFO] [launch.py:347:main] Process 3023907 exits successfully.
[2023-08-18 15:37:23,430] [INFO] [launch.py:347:main] Process 3023906 exits successfully.
```
And the model size is equal as the model which not use deepspeed MOQ algorithm like running the command by 
`python text-classification/run_glue.py \
  --model_name_or_path bert-base-cased \
  --task_name $TSK \
  --do_train \
  --do_eval \
  --max_seq_length 128 \
  --per_device_train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3 \
  --output_dir /tmp/$TSK/ \
  --fp16 \
  --warmup_steps 2`
Therefore I doubt it really uses the MOQ algorithm

Hello, can you raise the issue with the DeepSpeed team as this is not a problem with integration?

@pacman100 OK，thanks