Memory calculator for transformer models
Author: marksverdhei
Pull request: no

### Feature request

This feature request is quite high level. 
The feature is some tool, function, object, etc. that takes in information such as model config, trainer arguments, and max sequence length and calculates the expected memory usage. It could for instance be used to produce a warning if expected memory exceeds current available memory, and lets you select your hyperparameters with memory usage taken into account (instead of trying out which params raises memory errors and not). I mentioned this on the HF discord and was encouraged to make a feature request

### Motivation

When working with this library and the trainer API,
I've been missing some type of tool that can calculate the expected memory consumption of your model training.
`RuntimeError: CUDA error: out of memory` haunts us all and can perhaps be better understood if we're able to pre compute expected memory to see if the error is expected or not. It also makes it easier to select hyperparameters with memory constraints.


### Your contribution

Should this be of interest:
* how it will be integrated should probably be agreed upon first. 
* I'm willing to contribute to this in the summer if nobody has picked it up by then, should my help be wanted

Comments:
Reminds me of your current work on accelerate @muellerzr!

On my to do (asap) is to integrate that work from Accelerate. Should be in the next few weeks here. Basically how that one works is we retry the training loop reducing the batch size until we escape the CUDA OOM (this was request *many* times in our internal slack to integrate it here as well). Said implementation: https://github.com/huggingface/accelerate/blob/main/src/accelerate/memory_utils.py

Very interesting, I wasn't aware of this. Looking forward to the integration
Any thoughts on pre-calculating expected memory usage?
Or any reason why this would be unfeasible or impractical?

Without lots of very specific code, currently it is unfeasible (though won't be soon!).
The key is with pytorch's `meta` device. It currently doesn't work on all ops, but once it does we should be able to track all the sizes of the intermediate activation without real memory usage, getting us there. 

Otherwise we currently *could* by just doing the size of the model * the right number based on the optimizer selected, but we'd still be missing all of those intermediate activation sizes.

For now, the bs reducer is a good way to only add a few minutes (if not seconds) to get it going, hence why I went with that approach. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

FYI we've now built this: https://huggingface.co/spaces/hf-accelerate/model-memory-usage?logs=build

Thats awesome! Thanks a lot!

@muellerzr I'm having an error with the calculator. Is not working for me with "gated models". I'm getting this error:

Error
"Model `meta-llama/Llama-2-7b-chat-hf` had an error, please open a discussion on the model's page with the error message and name: `You are trying to access a gated repo.\nMake sure to request access at https://huggingface.co/meta-llama/Llama-2-7b-chat-hf and pass a token having permission to this repo either by logging in with `huggingface-cli login` or by passing `token=<your_token>`."

I'm using a working access token. I have been using it to download llama 2 models.

I would say there is part of the initial question that I would find useful but is not covered by accelerate's hf-space or the `accelerate.utils.memory.py` solution (iterative batch size reduction).

Namely the part about estimating memory usage according to the `max sequence length`.

The use case would be the ability to calculate the memory usage of a model, given an input sequence length, but for inference where it may not be feasible to iterative reduce the batch size. This way you could know before inference if a given sequence length is too large for the current compute.