How to run Trainer + DeepSpeed + Zero3 + PEFT 
Author: BramVanroy
Pull request: no
Labels: name: WIP

### System Info

- `transformers` version: 4.34.0.dev0
- Platform: Linux-5.14.0-284.25.1.el9_2.x86_64-x86_64-with-glibc2.34
- Python version: 3.11.4
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.3
- Accelerate version: 0.24.0.dev0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)


### Who can help?

 @ArthurZucker and @younesbelkada and @pacman100 and @muellerzr 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

[This script](https://gist.github.com/BramVanroy/f2abb3940111b73ae8923822ef6096dd) is a modification of the official run_clm script. The only additions are the BNB config and PEFT. Yet, I cannot get it to work with a [deepspeed zero3 config](https://github.com/philschmid/deep-learning-pytorch-huggingface/blob/main/training/configs/ds_falcon_180b_z3.json).

Requirements to install:

```
accelerate >= 0.12.0
torch >= 1.3
datasets >= 1.8.0
sentencepiece != 0.1.92
protobuf
evaluate
scikit-learn
trl
peft
bitsandbytes
```

In the past I have had issues with low_cpu_mem_usage but neither a true/false value seem to get this to work:

Command 1:

```sh
deepspeed --include="localhost:0,1" run_clm.py \
   --model_name_or_path facebook/opt-125m\
  --dataset_name wikitext\
  --dataset_config_name wikitext-2-raw-v1\
  --per_device_train_batch_size 2\
  --per_device_eval_batch_size 2\
  --do_train\
  --do_eval\
  --output_dir /tmp/test-clm\
  --deepspeed deepspeed_configs/ds_config_zero3.json\
  --low_cpu_mem_usage true
```
==> `ValueError: DeepSpeed Zero-3 is not compatible with `low_cpu_mem_usage=True` or with passing a `device_map`.`

Command 2:

```sh
deepspeed --include="localhost:0,1" run_clm.py \
   --model_name_or_path facebook/opt-125m\
  --dataset_name wikitext\
  --dataset_config_name wikitext-2-raw-v1\
  --per_device_train_batch_size 2\
  --per_device_eval_batch_size 2\
  --do_train\
  --do_eval\
  --output_dir /tmp/test-clm\
  --deepspeed deepspeed_configs/ds_config_zero3.json\
  --low_cpu_mem_usage false
```

==> `ValueError: weight is on the meta device, we need a `value` to put in on 0.`

### Expected behavior

Any option to make this combination of Trainer + DeepSpeed + Zero3 + PEFT work.

Comments:
Hello @BramVanroy, DeepSpeed is not compatible with BitsandBytes. You can either use Trainer + PEFT + DeepSpeed ZeRO3 or Trainer + PEFT + BitsandBytes/int4/int8. Above, you are trying to do Trainer + PEFT + BitsandBytes 4-bit + DeepSpeed ZeRO3 which isn't supported. For using PEFT with DeepSpeed ZeRO3, please refer this nice blog https://www.philschmid.de/deepspeed-lora-flash-attention

Are there plans to improve this situation? Or do you have a starting point where I can help? Having Zero3 support in this case would be very useful.

Bump to keep alive

OMG this thread will save my sanity. Been digging at this for days and thought something was wrong on my end. 

I am trying to run lora/qlora with Zero3 through the Axolotl library and I am encountering the exact same issue. 

It would be extremely useful if we can run 4bit with Zero3. From my understanding, Zero3 is the only way to get model parallelism with DeepSpeed. This would allow us to finetune larger models on multiple GPUs with smaller vram limits. At the moment, it is only possible to run naive model parallelism (only 1 GPU would be active at any given time although the larger models can be spread across multiple GPUs).  

Please let me know if there is an update on this issue.

> OMG this thread will save my sanity. Been digging at this for days and thought something was wrong on my end.
> 
> I am trying to run lora/qlora with Zero3 through the Axolotl library and I am encountering the exact same issue.
> 
> It would be extremely useful if we can run 4bit with Zero3. From my understanding, Zero3 is the only way to get model parallelism with DeepSpeed. This would allow us to finetune larger models on multiple GPUs with smaller vram limits. At the moment, it is only possible to run naive model parallelism (only 1 GPU would be active at any given time although the larger models can be spread across multiple GPUs).
> 
> Please let me know if there is an update on this issue.

Did you manage to solve this issue?

Will the issue be solved after deepspeed release the following information?
https://www.deepspeed.ai/tutorials/MoQ-tutorial/