device_map='auto' causes memory to not be freed with torch.cuda.empty_cache()
Author: oobabooga
Pull request: no

### System Info

If I load a model like this

```
model = AutoModelForCausalLM.from_pretrained("models/opt-13b", device_map='auto', load_in_8bit=True)
```

and then do

```
model = None
torch.cuda.empty_cache()
```

the VRAM is not freed. The only way I have found to release it is to kill the program and start it over. 

Freeing the memory like this works if the model is loaded the normal way with

```
model = AutoModelForCausalLM.from_pretrained("models/opt-13b", low_cpu_mem_usage=True, torch_dtype=torch.float16).cuda()
```


@ArthurZucker @younesbelkada

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Code snippet is provided in the description above.

### Expected behavior

VRAM should be freed with `torch.cuda.empty_cache()`.

Comments:
That is an interesting bug, but should probably be adressed in `accelerate` as using `device_map = "auto"` is reliant on the accelerate library. 

I think it's the 8bit part that may be causing the issue, actually ;-) 

I was able to reproduce the issue with `accelerate` loaded models (without 8-bit):
```
import torch
from transformers import AutoModelForCausalLM

model_8bit = AutoModelForCausalLM.from_pretrained("facebook/opt-350m", device_map="auto", load_in_8bit=True)
model_accelerate = AutoModelForCausalLM.from_pretrained("facebook/opt-350m", device_map="auto", torch_dtype=torch.float16)
model_torch = AutoModelForCausalLM.from_pretrained(
    "facebook/opt-350m", low_cpu_mem_usage=True, torch_dtype=torch.float16
).cuda()

del model_accelerate
torch.cuda.empty_cache()

del model_8bit
torch.cuda.empty_cache()

del model_torch
torch.cuda.empty_cache()
```
With this script the GPU VRAM is freed only after the lines:
```
del model_torch
torch.cuda.empty_cache()
```
I also profiled the GPU memory and observed that the allocated memory decreases after the aforementioned line. 
<img width="1134" alt="Screenshot 2023-01-16 at 11 06 14" src="https://user-images.githubusercontent.com/49240599/212652122-796186e4-c476-42b6-b628-7def9d7ea3d0.png">
Note that the 8bit Linear modules seems to behave correctly with respect to `torch.cuda.empty_cache`, i.e:
```
import torch
from transformers import AutoModelForCausalLM
import bitsandbytes as bnb

model_8bit = AutoModelForCausalLM.from_pretrained("facebook/opt-350m", device_map="auto", load_in_8bit=True)
model_accelerate = AutoModelForCausalLM.from_pretrained("facebook/opt-350m", device_map="auto", torch_dtype=torch.float16)
model_torch = AutoModelForCausalLM.from_pretrained(
    "facebook/opt-350m", low_cpu_mem_usage=True, torch_dtype=torch.float16
).cuda()

linear_8bit = bnb.nn.Linear8bitLt(10000, 10000).to("cuda")

del model_accelerate
torch.cuda.empty_cache()

del model_torch
torch.cuda.empty_cache()

del model_8bit
torch.cuda.empty_cache()

del linear_8bit
torch.cuda.empty_cache()
```
The VRAM goes correctly down after the line:
```
del linear_8bit
torch.cuda.empty_cache()
```
So the issue should be on `accelerate` side but not sure where exactly, will investigate more but if you have more insights @sgugger @muellerzr would love to hear from you!

I have tried and can indeed reproduce without the 8bit loading. I don't know why the cache appears nonempty, but iterating on a loop (re-creating the model and then deleting it several times) does not result in a memory increase, so the memory is reused when we need it.

If someone manages to find more about this, I'd be very interested to learn why this is the case.

```
gc.collect()
torch.cuda.empty_cache()
gc.collect()
```
may free memory.

I can confirm that ~690MB of seems to be freed (monitoring with `nvidia-smi`) - which corresponds to the size of the weights of opt-350 in fp16 thanks to the trick proposed by @git-longcat 
```
import torch
import gc
from transformers import AutoModelForCausalLM

model_accelerate = AutoModelForCausalLM.from_pretrained("facebook/opt-350m", device_map={"":0}, torch_dtype=torch.float16)
model_torch = AutoModelForCausalLM.from_pretrained(
    "facebook/opt-350m", low_cpu_mem_usage=True, torch_dtype=torch.float16
).cuda()

del model_accelerate
gc.collect()
torch.cuda.empty_cache()

del model_torch
torch.cuda.empty_cache()
```
@oobabooga can you try on your side and let us know if this trick works? 
It seems crucial to call `gc.collect()` before `torch.cuda.empty_cache()` and not after 

@git-longcat @younesbelkada  I confirm that this works. Thank you. 

In fact, all I needed was

```
model = None
gc.collect()
torch.cuda.empty_cache()

```

This doesn't work:

```
model = None
torch.cuda.empty_cache()
gc.collect()
```

This resolves the issue for me. I don't know if it can still be considered a bug or not.

@younesbelkada @oobabooga in Accelerate dev now (install via `pip install git+https://github.com/huggingface/accelerate`) I've introduced a `release_memory` util that will perform the above for `n` objects easily:
```python
  >>> import torch
  >>> from accelerate.utils import release_memory

  >>> a = torch.ones(1000, 1000).cuda()
  >>> b = torch.ones(1000, 1000).cuda()
  >>> a, b, model = release_memory(a, b, model)
```
Just ensure that the objects are in the same order they were passed in as so the memory will be fully overriden. (see https://github.com/huggingface/accelerate/pull/990 for more information)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.