Running out of memory while finetuning and inferencing VideoMAE due to which script is being killed. 
Author: pramodsarvi
Pull request: no

### System Info

accelerate==0.30.1
aiohttp==3.9.5
aiosignal==1.3.1
async-timeout==4.0.3
attrs==23.2.0
av==12.0.0
certifi==2024.2.2
charset-normalizer==3.3.2
datasets==2.19.1
dill==0.3.8
evaluate==0.4.2
filelock==3.14.0
frozenlist==1.4.1
fsspec==2024.3.1
fvcore==0.1.5.post20221221
huggingface-hub==0.23.0
idna==3.7
iopath==0.1.10
Jinja2==3.1.4
joblib==1.4.2
MarkupSafe==2.1.5
mpmath==1.3.0
multidict==6.0.5
multiprocess==0.70.16
networkx==3.1
numpy==1.24.4
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==8.9.2.26
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-nccl-cu12==2.20.5
nvidia-nvjitlink-cu12==12.4.127
nvidia-nvtx-cu12==12.1.105
packaging==24.0
pandas==2.0.3
parameterized==0.9.0
pillow==10.3.0
portalocker==2.8.2
psutil==5.9.8
pyarrow==16.1.0
pyarrow-hotfix==0.6
python-dateutil==2.9.0.post0
pytorchvideo==0.1.5
pytz==2024.1
PyYAML==6.0.1
regex==2024.5.15
requests==2.31.0
safetensors==0.4.3
scikit-learn==1.3.2
scipy==1.10.1
six==1.16.0
sympy==1.12
tabulate==0.9.0
termcolor==2.4.0
threadpoolctl==3.5.0
tokenizers==0.19.1
torch==2.3.0
torchaudio==2.3.0
torchvideo==0.0.0
torchvision==0.18.0
tqdm==4.66.4
transformers==4.41.0
triton==2.3.0
typing_extensions==4.11.0
tzdata==2024.1
urllib3==2.2.1
xformers==0.0.26.post1
xxhash==3.4.1
yacs==0.1.8
yarl==1.9.4


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run VideoMAE notebook

### Expected behavior

Is there any way to reduce system RAM usage (I have 32 GB system RAM/ 25 GB NVRAM)

Comments:
Hi @pramodsarvi, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

Hi @pramodsarvi you can do two main things for it, use mixed precision training or try to reduce the `batch_size`. Mixed precision training can be done with torch.autocast and this context manager can automatically be used in the notebook through `bf16=True` argument in `TrainingArguments`.
And rest I think can be discussed in the forums as said above.

Cheers!

GPU usage is stable but system RAM is exploding exponentially

yeah it is due to the high memory demands of storing and processing these frames. You can try if above solution anyhow helps otherwise here is a [link](https://discuss.tensorflow.org/t/running-out-of-memory-while-performing-model-training/21644/5) I am sharing which could provide further general details. And also sometimes GPU doesn't get fully utilised during training and that data don't get transferred efficiently between the CPU and GPU.

The issue still persists after modifying. I'll do checkout the link.
Thanks @RUFFY-369 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @pramodsarvi , the above bot just reminded me of this issue and an alternate solution came to my mind which didn't come previously as I was busy with work. If the problem is still there then you can check out the below solution:
```
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
    "Trelis/Llama-2-7b-chat-hf-function-calling-v2", low_cpu_mem_usage=True,offload_state_dict=True
)
```
`low_cpu_mem_usage` : it will load the model using ~1x model size CPU memory
`offload_state_dict` : it will temporarily offload the CPU state dict to hard drive and will prevent getting out of RAM

Cheers!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.