Memory Leak When Using padding="max_length" in T5 Text Encoder On CPU
Author: CyberVy
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.45.1
- Platform: Linux-6.6.56+-x86_64-with-glibc2.35
- Python version: 3.10.14
- Huggingface_hub version: 0.25.1
- Safetensors version: 0.4.5
- Accelerate version: 0.34.2
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.4.0 (True)
- Tensorflow version (GPU?): 2.16.1 (True)
- Flax version (CPU?/GPU?/TPU?): 0.8.4 (gpu)


### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from transformers import T5EncoderModel,T5Tokenizer


def get_t5_prompt_embeds(tokenizer_2, text_encoder_2,prompt = None,max_sequence_length = 512):

        prompt = [prompt] if isinstance(prompt, str) else prompt

        text_inputs = tokenizer_2(prompt,padding="max_length",max_length=max_sequence_length,
            truncation=True,return_length=False,return_overflowing_tokens=False,return_tensors="pt",
        )
        text_input_ids = text_inputs.input_ids
        prompt_embeds = text_encoder_2(text_input_ids, output_hidden_states=False)[0]

        return prompt_embeds


tokenizer = T5Tokenizer.from_pretrained(...)
encoder = T5EncoderModel.from_pretrained(...)
get_t5_prompt_embeds(tokenizer, encoder, 'hello')
```
When the padding="max_length" parameter is passed to the tokenizer, the memory usage increases significantly, leading to a CPU memory leak of approximately 5GB. Upon removing the padding="max_length" parameter, the memory leak disappears, and the previously leaked memory is recovered.

### Expected behavior

The function should generate the embeddings for the provided prompt without causing a significant increase in memory usage.

Comments:
```python
def get_t5_prompt_embeds(tokenizer_2, text_encoder_2,prompt = None,max_sequence_length = 512):

        prompt = [prompt] if isinstance(prompt, str) else prompt
        with torch.no_grad():
            text_inputs = tokenizer_2(prompt,padding="max_length",max_length=max_sequence_length,
                truncation=True,return_length=False,return_overflowing_tokens=False,return_tensors="pt",
            )
            text_input_ids = text_inputs.input_ids
            prompt_embeds = text_encoder_2(text_input_ids, output_hidden_states=False)[0]
    
            return prompt_embeds
```
After further investigation, I found that wrapping the code in torch.no_grad() resolves the issue and releases the leaked memory. However, I believe this should be handled automatically by the T5 Encoder itself, as this behavior could be unintuitive for users.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey! Thanks for updating, by default we do like torch: model is computing grads when ran like this!