T5EncoderModel slower in half-precision
Author: DA-L3
Pull request: no

Hi, 

I am encountering troubles in understanding why the half-precision version of the T5Encoder infers slower than the full-precision one. 

## To reproduce
Starting with the `half`-precision. 
```python
import torch
from transformers import T5EncoderModel, T5Tokenizer
import time

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
seq="Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet. Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. At vero eos et accusam et justo duo dolores et ea rebum. Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet."
seq = ' '.join([seq] * 50)

model = T5EncoderModel.from_pretrained('t5-large', cache_dir='model').to(device)
model = model.half()
model = model.eval()

tokenizer = T5Tokenizer.from_pretrained('t5-large', cache_dir='model')

token_timer = time.time()
tokens = tokenizer.batch_encode_plus(seq, add_special_tokens=True, padding='longest', return_tensors='pt')
end_token = time.time()
input_ids = tokens['input_ids'].to(device)
attention_mask = tokens['attention_mask'].to(device)

model_timer = time.time()
with torch.no_grad():
    ignored = model(input_ids=input_ids,attention_mask=attention_mask)
end_timer = time.time()

print(f'Full process:\t{end_timer - token_timer}')
print(f'Model only:\t{end_timer - model_timer}')
print(f'Token only:\t{end_token - token_timer}')
``` 
To use the `full`-precision, just drop the `model = model.half()` line.

## The output
The `half`-precision:
```
Full process:	10.929700136184692
Model only:	3.4116599559783936
Token only:	7.5169923305511475
```
The `full`-precision:
```
Full process:	7.794144153594971
Model only:	0.23117947578430176
Token only:	7.562213897705078
```
First, I would expect that the half-precision model is faster but secondly what is more confusing to me is the time difference in `Model only`, which measures the time needed to execute the the `torch.no_grad()`-part. 
Is there an implementation problem in the code snippet?

Comments:
@stas00 or @sgugger can chime in if I'm wrong, but I believe half-precision performance improvement is strongly tied to hardware: even hardware that handles half-precision like pascal GPUs may not see a speed increase with FP16 compared to FP32, and I believe it can have the opposite effect. Could you share your setup?

You can check this thread for a similar question: https://github.com/huggingface/transformers/issues/9179

I don't think we have resolved this conundrum in https://github.com/huggingface/transformers/issues/9179 - it got closed w/o a resolution.

Running your test on 2 cards:

```
rtx-3090

fp16:

Full process:   7.10092830657959
Model only:     1.9809677600860596
Token only:     5.1195228099823

fp32:

Full process:   5.614374399185181
Model only:     0.6039936542510986
Token only:     5.009963750839233

gtx-1070

fp16:

Full process:   17.52509307861328
Model only:     12.342488050460815
Token only:     5.182169198989868

fp32:

Full process:   5.362875461578369
Model only:     0.3538181781768799
Token only:     5.008580923080444
```

This investigation most likely will require using torch profiler to get to the root of it.

Thank you for answering and reffering to the other issue, since it seems to be an ongoing mystery, I will close this issue w/o resolution for now.