`device_map="auto"` doesn't use all available GPUs when `load_in_8bit=True`
Author: yukw777
Pull request: no

### System Info

- `transformers` version: 4.28.0.dev0
- Platform: Linux-4.18.0-305.65.1.el8_4.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.4
- Huggingface_hub version: 0.13.3
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes

### Who can help?

@sgugger @kooshi 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

On a machine with more than 2 GPUs (I have 4*A40s)
```python
model = transformers.LlamaForCausalLM.from_pretrained(
    "path/to/converted/llama-65B",
    load_in_8bit=True,
    device_map="auto"
)
```
You'll see that only the first two GPUs are filled up. Possibly related to https://github.com/huggingface/transformers/pull/22377.

### Expected behavior

All 4 GPUs should get parameters.

Comments:
If I specify `max_memory`, the parameters do get distributed according to it.

```python
model = transformers.LlamaForCausalLM.from_pretrained(
    "path/to/converted/llama-65B",
    load_in_8bit=True,
    device_map="auto",
    max_memory={0: "10GB", 1: "10GB", 2: "48GB", 3: "48GB"}
)
```

cc @younesbelkada 

You can also specify `device_map="balanced"` to get the parameters evenly dispatched.

Hmm maybe this is unrelated to `load_in_8bit`, can you try without that and let us know?
on the other hand I second what @Xmaster6y said, you can use `balanced` in this case

balanced and auto are the same thing FYI.

Yup, I knew `auto` and `balanced` were the same, but tried `balanced` for good measure. Same behavior :/.

I just verified that without `load_in_8bit`, the parameters are distributed evenly among the GPUs as expected.

So this is specifically for `load_in_8bit`. I think @younesbelkada made a fix after the last patch. Could you try an install from source?

I already installed from source.

```
$ pip freeze | grep transformers
transformers @ git+https://github.com/huggingface/transformers.git@15641892985b1d77acc74c9065c332cd7c3f7d7f
```

I can't dig too deeply into this until later, and I don't have more than 2 GPUs to test, but I can say that the actual size calculations and dispatch are all done in [accelerate](https://github.com/huggingface/accelerate), and the calculation changed as little as 3 weeks ago, so make sure you have the latest installed.

If that doesn't fix it, and you want to dig into it, I'd recommend just sprinkling some `print`s around the relevant `accelerate` and `transformers` functions, just to get some visibility into what it thinks it's calculating.

It'll be somewhere in
`transformers/modeling_utils.py` which calls
`get_balanced_memory` and `infer_auto_device_map` in `accelerate/utils/modeling.py`


Thanks @kooshi! It seems like something was fixed since accelerate 0.8. Installing accelerate from source resolved this issue.

I'm still running into this

If you want all GPUs to be used I think you should probably use `device_map = "sequential"`