load_in_4bit seems dosen't work as experted which actually increase GPU memory usage when using zero3 via accelerate
Author: tqjack
Pull request: no

### System Info

transformers==4.31.0
deepspeed==0.9.2
peft==0.4.0
bitsandbytes==0.39.0
torch==1.13.0
CUDA Version 11.8
GPUS 8x A100 80gb

### Who can help?

@pacman100

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I compared the from_pretrained method w/o load_in_4bit passed in and I found out that after passing load_in_4bit in from_pretrained method I can't load model with same hardware and same accelerate config.

accelerate config:
ds_zero3.yaml
```
compute_environment: LOCAL_MACHINE
deepspeed_config:
  gradient_accumulation_steps: 1
  gradient_clipping: 1.0
  offload_optimizer_device: 'none'
  offload_param_device: 'cpu'
  zero3_init_flag: true
  zero3_save_16bit_model: true
  zero_stage: 3
distributed_type: DEEPSPEED
downcast_bf16: 'no'
machine_rank: 0
main_training_function: main
mixed_precision: bf16
num_machines: 1
num_processes: 8
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false
```

load in normal way:
test.py
```import torch
from accelerate import Accelerator

from transformers import (
    AutoModelForCausalLM,
)

def main():
    accelerator = Accelerator()
    model_name_or_path = "local path of sambanovasystems/BLOOMChat-176B-v1"

    model = AutoModelForCausalLM.from_pretrained(
        model_name_or_path,
        trust_remote_code=True)

if __name__ == "__main__":
    main()
```
run command:
```
accelerate launch --config_file ds_zero3.yaml test.py
```
it works just fine, 
GPU memory usage: 22GB x 8 
peak CPU memory usage: ~500GB

then, I try to use load_in_4bit=True
test_4bit.py
```import torch
from accelerate import Accelerator

from transformers import (
    AutoModelForCausalLM,
)

def main():
    accelerator = Accelerator()
    model_name_or_path = "local path of sambanovasystems/BLOOMChat-176B-v1"

    model = AutoModelForCausalLM.from_pretrained(
        model_name_or_path,
        trust_remote_code=True,
        load_in_4bit=True)

if __name__ == "__main__":
    main()
```
run command:
```
accelerate launch --config_file ds_zero3.yaml test_4bit.py
```
OOM error, it seems nether zero3 nor parameters offlaod working as experted. peak cpu usage ~500GB in this case.

also tried 
```
quantization_config = BitsAndBytesConfig(
   load_in_4bit=True,
   bnb_4bit_quant_type="nf4",
   bnb_4bit_use_double_quant=True,
   bnb_4bit_compute_dtype=torch.bfloat16
)
```
OOM

### Expected behavior

I am trying to finetune bloomchat176b using qlora and qlora should using less hardware resource. Thanks for help

Comments:
Hello, QLoRA and DeepSpeed can't be used together. DeepSpeed doesn't work with quantized parameters.

> Hello, QLoRA and DeepSpeed can't be used together. DeepSpeed doesn't work with quantized parameters.

is that means I can't do zero optimization while using qlora, at least for now?  ddp is the only parallelism method compatible with qlora?

Hello, yes

cc @younesbelkada for adding more context just in case

Hi there!
Indeed 4bit + 8bit is not supported with DS, let's maybe add that check on accelerate.preprare (I can work on that)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.