Loading of a model takes much RAM, passing to CUDA doesn’t free RAM
Author: Artyrm
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.10.0.dev0
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.11
- PyTorch version (GPU?): 1.7.0+cu110 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
- benchmarks: @patrickvonplaten
- pipelines: @LysandreJik

## Information

Model I am using: EleutherAI/gpt-neo-1.3B

The problem arises when using:
* my own modified scripts: (give details below)

## To reproduce

Steps to reproduce the behavior:
[Google Colaboratory](https://colab.research.google.com/drive/1qptTsxuRvxnTq2FI39a9p8VSquH5Qafl?usp=sharing) notebook
You will need “Large memory” instance, since while transferring to CUDA it even overshoots 13GB RAM limit. I use Torch 1.7.0+cu110 since instance has CUDA 11.2. But with the default 1.9.0+cu102 it is more or less the same.

I’m trying to finetune 1.3B model. And so I search for the way to optimize RAM usage (to be able to use cpu_offload with deep_speed).
I noted that after load of a model it takes much RAM.

![image](https://user-images.githubusercontent.com/21180686/130306142-402e8aec-a7d9-48ca-bbbd-6b6189294c77.png)

After model is loaded:
11.51 GB total memory used
0.0 GB used by torch objects on GPU
2 MiB total mem used on GPU

And when I move it to GPU it
a) takes only 5GB in VRAM (perhaps another 1.3GB is taken by Torch).
b) doesn’t free any RAM, even takes some 2.5GB more.

So the problem I see:
a) Model occupates much more space in RAM then in VRAM.
b) It doesn’t free RAM upon moving to CUDA.

Python garbage collector doesn’t help also.

Any thoughts on this?

Comments:
Hey @Artyrm,

It's quite difficult for us to reproduce the error - could you post a link to a google colab where we can rerun the code? Also do you see the same behavior locally or is it just on google colab?

Hi, @patrickvonplaten.
I have provided the link in my post, at the beggining of "reproduce" section.

And yes, I can see it locally too, though I have only 2GB local VRAM, so I have to use small models, and memory waste is less obvious.

Related I think: https://github.com/huggingface/transformers/pull/12106#discussion_r649876604

@patil-suraj - we wanted to change GPT-Neo to use a local attention mask instead of the local attention layers as it was shown to be faster and less memory intensive no? Should we tackle that again?

Also related: https://github.com/huggingface/transformers/pull/11736

Need to say, that I have same problem locally (in less scale, since much less VRAM available) with other models, for example `sberbank-ai/rugpt3medium_based_on_gpt2` Although it is also "GPT-3-like" model.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

That's a pity.
I hoped someone would have some ideas about it.

@patil-suraj - did you look into this by any chance? 

Similar to unresolved issue https://github.com/huggingface/transformers/issues/13624

I tried some experiments, and it seems it's related to PyTorch rather than Transformers model.
It seems that when a model is moved to GPU, all CPU RAM is not immediately freed, as you could see in this [colab](https://colab.research.google.com/drive/1FvUtyCXFfx1cMexO24IvXXrkLPTt_Ok5?usp=sharing), but you could still use the RAM to create other objects, and it'll then free the memory or you could manually  call `gc.collect`.

Also note that, `py.memory_info()[0]` gives total memory used by the process and not the current memory in use. We could use the `psutil.virtual_memory().available` to get the available RAM. I've used it in the colab above so you could see the difference.

Also gently pinging @stas00 who might be able to shed some light here :) 
 

It's most likely a python issue and not torch's - this is because `gc.collect()` is a scheduled event and doesn't always run when a large object is freed.

You can read more about it here: https://docs.python.org/3/library/gc.html

You can experiment with setting a lower threshold https://docs.python.org/3/library/gc.html#gc.set_threshold

I don't think there is any harm in `transformers` calling `gc.collect` immediately after switching the model to gpu - it'll be run anyway sooner or later, and thus it's not like it'll be introducing a performance hit at that particular point. 

Wrt debug/tracing memory usage in notebooks I highly recommend using https://github.com/stas00/ipyexperiments since it prints out all that memory usage automatically for you after each cell is run, so it's much easier to run. if using on colab there are some nuances to handle:
https://github.com/stas00/ipyexperiments#google-colab

Hmm, but actually `ipyexperiments` calls `gc.collect()` by itself to measure things correctly, so it's going to hide this issue. So probably scratch that idea in this particular context.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.