Memory issue in Transformers
Author: AishwaryaVerma
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->
Hello everyone, 
I am using distilBert from hugging face transformers and loading its tokenizer and model in my class. I have already quantised it to reduce the size of the model but still this class is taking nearly 500 MB of the memory though the model is taking only 100 MB. Then I looked into github repo of HuggingFace Transformers they are not using __all__ and __slots__ to their classes and functions, to reduce the memory size of the classes. My question is how do I reduce the memory size while loading any transformer model and why hugging face is not using __all__ and __slots__ in their codebase to make it more efficient.

Thank you in advance.  
<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Comments:
Do you have a paper/link about those "add" and "slots" that you mention? I have never heard of it.

> Do you have a paper/link about those "add" and "slots" that you mention? I have never heard of it.
Sorry,  it was __all__.
Please check the documentation link.
https://docs.python.org/3/tutorial/modules.html#importing-from-a-package
https://docs.python.org/3/reference/datamodel.html#slots

> Do you have a paper/link about those "add" and "slots" that you mention? I have never heard of it.

Some stackoverflow links:
https://stackoverflow.com/questions/44834/can-someone-explain-all-in-python
https://stackoverflow.com/questions/472000/usage-of-slots
https://stackoverflow.com/questions/14118564/how-does-slots-avoid-a-dictionary-lookup/14119024#14119024

I thought you meant some kind of deep learning optimization. I am curious to see the impact of slots. I guess it could be useful to have a closer look at how much it decreases memory usage. _However_ it seems highly unlikely that you will get the consumption down by a lot. When I read through that top answer, the memory that you save is in the _bytes_, not even kilobytes, let alone hundreds of megabytes. 

If you want, you can rewrite parts of transformers and benchmark whether you'll find a difference, but I doubt it.

Other things that may help:

- use evaluation mode and no_grad
- trace your model
- use something like ONNX to improve inference

> I thought you meant some kind of deep learning optimization. I am curious to see the impact of slots. I guess it could be useful to have a closer look at how much it decreases memory usage. _However_ it seems highly unlikely that you will get the consumption down by a lot. When I read through that top answer, the memory that you save is in the _bytes_, not even kilobytes, let alone hundreds of megabytes.
> 
> If you want, you can rewrite parts of transformers and benchmark whether you'll find a difference, but I doubt it.
> 
> Other things that may help:
> 
>     * use evaluation mode and no_grad
> 
>     * trace your model
> 
>     * use something like ONNX to improve inference

Ok. Thank you.

> this class is taking nearly 500 MB of the memory though the model is taking only 100 MB

@AishwaryaVerma which memory do you mean when you're speaking about 100 MB? Does your model take 100 MB disk space but the loaded one takes 500 MB RAM?

Sorry for replying yo

> 
> 
> > this class is taking nearly 500 MB of the memory though the model is taking only 100 MB
> 
> @AishwaryaVerma which memory do you mean when you're speaking about 100 MB? Does your model take 100 MB disk space but the loaded one takes 500 MB RAM?

Sorry for replying you late. But I was talking about RAM.