MT5 OutOfMemory Error 
Author: semindan
Pull request: no

### System Info

## Setup 1
- `transformers` version: 4.32.1
- Platform: Linux-4.18.0-425.13.1.el8_7.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.3
- Accelerate version: 0.22.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

## Setup 2
- `transformers` version: 4.31.0
- Platform: Linux-4.18.0-425.13.1.el8_7.x86_64-x86_64-with-glibc2.28
- Python version: 3.11.4
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.2
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Have a GPU (I'm using 1 GPU) and a correct installation of pytorch on your system (`torch.cuda.is_available()` should be `True`) 
2.  Copy the official script from [mt5 repo](https://huggingface.co/google/mt5-base), you can even throw out the tokenizer lines:
``` python
# Load model directly
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("google/mt5-base")
model = AutoModelForSeq2SeqLM.from_pretrained("google/mt5-base")
```
3. Execute the script

This gives me OOM, sometimes it gets stuck and then killed, sometimes it hangs for a long period of time and I just shut it down.
## Setup 1
```bash
semindan@g06 ~/mtl_thesis (dev)> time python                                                                                                            (mtl_t) 
```
```python
Python 3.10.12 | packaged by conda-forge | (main, Jun 23 2023, 22:40:32) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> # Load model directly
>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
oTokenizer.from_pretrained("google/mt5-base")
model = AutoModelForSeq2SeqLM.from_pretrained("google/mt5-base")
>>> 
>>> tokenizer = AutoTokenizer.from_pretrained("google/mt5-base")
You are using the default legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. If you see this, DO NOT PANIC! This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=True`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
/home/semindan/.conda/envs/env_mamba/envs/mtl_t/lib/python3.10/site-packages/transformers/convert_slow_tokenizer.py:470: UserWarning: The sentencepiece tokenizer that you are converting to a fast tokenizer uses the byte fallback option which is not implemented in the fast tokenizers. In practice this means that the fast version of the tokenizer can produce unknown tokens whereas the sentencepiece version would have converted these unknown tokens into a sequence of byte tokens matching the original piece of text.
  warnings.warn(
>>> model = AutoModelForSeq2SeqLM.from_pretrained("google/mt5-base")
```
```bash
^Z
________________________________________________________
Executed in  117.68 secs      fish           external
   usr time  334.00 micros  334.00 micros    0.00 micros
   sys time  304.00 micros  304.00 micros    0.00 micros

fish: Job 1, 'time python' has stopped
semindan@g06 ~/mtl_thesis (dev)> fish: Job 1, 'time python' terminated by signal SIGKILL (Forced quit)                                                  (mtl_t) 
semindan@g06 ~/mtl_thesis (dev)> exit                                                                                                                   (mtl_t) 
slurmstepd: error: Detected 1 oom_kill event in StepId=6242010.0. Some of the step tasks have been OOM Killed.
srun: error: g06: task 0: Out Of Memory
```
## Setup 2
``` python
Python 3.11.4 | packaged by conda-forge | (main, Jun 10 2023, 18:08:17) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> # Load model directly
>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
oTokenizer.from_pretrained("google/mt5-base")
model = AutoModelForSeq2SeqLM.from_pretrained("google/mt5-base")
>>> 
>>> tokenizer = AutoTokenizer.from_pretrained("google/mt5-base")
You are using the legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. This means that tokens that come after special tokens will not be properly handled. We recommend you to read the related pull request available at https://github.com/huggingface/transformers/pull/24565
/home/semindan/.conda/envs/env_mamba/envs/llm_env/lib/python3.11/site-packages/transformers/convert_slow_tokenizer.py:470: UserWarning: The sentencepiece tokenizer that you are converting to a fast tokenizer uses the byte fallback option which is not implemented in the fast tokenizers. In practice this means that the fast version of the tokenizer can produce unknown tokens whereas the sentencepiece version would have converted these unknown tokens into a sequence of byte tokens matching the original piece of text.
  warnings.warn(
>>> model = AutoModelForSeq2SeqLM.from_pretrained("google/mt5-base")
```
```bash
^C^C^C^C^C^C^C^Z
________________________________________________________
Executed in  577.45 secs      fish           external
   usr time  452.00 micros  452.00 micros    0.00 micros
   sys time    0.00 micros    0.00 micros    0.00 micros

fish: Job 1, 'time python' has stopped
semindan@g10 ~/mtl_thesis (dev)> fish: Job 1, 'time python' terminated by signal SIGKILL (Forced quit)                                                (llm_env) 
semindan@g10 ~/mtl_thesis (dev)> exit                                                                                                                 (llm_env) 
slurmstepd: error: Detected 8 oom_kill events in StepId=6242005.0. Some of the step tasks have been OOM Killed.
srun: error: g10: task 0: Out Of Memory
```

### Expected behavior

I expect the model to load. When no GPU is present, the script behaves as expected:
```bash
(mtl_t) python
```
```python
Python 3.10.12 | packaged by conda-forge | (main, Jun 23 2023, 22:40:32) [GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> # Load model directly
>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
oTokenizer.from_pretrained("google/mt5-base")
model = AutoModelForSeq2SeqLM.from_pretrained("google/mt5-base")
>>> 
>>> tokenizer = AutoTokenizer.from_pretrained("google/mt5-base")
You are using the default legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. If you see this, DO NOT PANIC! This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=True`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
/home/semindan/.conda/envs/env_mamba/envs/mtl_t/lib/python3.10/site-packages/transformers/convert_slow_tokenizer.py:470: UserWarning: The sentencepiece tokenizer that you are converting to a fast tokenizer uses the byte fallback option which is not implemented in the fast tokenizers. In practice this means that the fast version of the tokenizer can produce unknown tokens whereas the sentencepiece version would have converted these unknown tokens into a sequence of byte tokens matching the original piece of text.
  warnings.warn(
>>> model = AutoModelForSeq2SeqLM.from_pretrained("google/mt5-base")
Downloading (…)neration_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████| 147/147 [00:00<00:00, 238kB/s]
>>> 
```

Comments:
I can't reproduce this, and not sure if I see anything wrong here. Would recommend you to use `low_cpu_mem_usage = True` as whether you have a GPU or not should not affect the scripts you shared (you are not using `to` or `.cuda()` anywhere. `device_map = "auto"` should help if you don't have enough RAM, and want  to load on GPU. In the snippets you are not loading on GPU anyway. 

yep, the problem was the low amount of CPU RAM on the node :skull:, with more RAM it behaves as expected, thank you