T5EncoderModel Runtime Error: Scalar type Half but found Float
Author: sacdallago
Pull request: no

### System Info

- `transformers` version: 4.26.0
- Platform: Linux-5.4.0-124-generic-x86_64-with-glibc2.10
- Python version: 3.8.0
- Huggingface_hub version: 0.12.0
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Y (V100-DGXS; Driver Version: 525.85.12 ; CUDA Version: 12.0 )
- Using distributed or parallel set-up in script?: N

### Who can help?

@ArthurZucker and @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Created a new conda env from python==3.8
2. Installed latest dependcies via pip without version tag: `pip install numpy torch transformers accelerate tqdm sentencepiece`
3. Ran the following script:

```
import torch

from transformers import T5EncoderModel, T5Tokenizer

transformer_name = "Rostlab/prot_t5_xl_half_uniref50-enc"
prott5_dir =  "checkpoints/prott5"
device = "cuda"

model = T5EncoderModel.from_pretrained(transformer_name, torch_dtype=torch.float16, cache_dir=prott5_dir, output_attentions=False)
model = model.to(device)
model = model.eval()

tokenizer = T5Tokenizer.from_pretrained(transformer_name, do_lower_case=False, cache_dir=prott5_dir)

token_encoding = tokenizer.batch_encode_plus(["SEQVENCE", "SEQQENCE"], add_special_tokens=True, padding='longest')

input_ids = torch.tensor(token_encoding['input_ids']).to(device)
attention_mask = torch.tensor(token_encoding['attention_mask']).to(device)


model(input_ids, attention_mask=attention_mask)
```

4. Changing `input_ids` and `attention_mask` to directly consume the dicts items (e.g. `input_ids = token_encoding['input_ids']`) leads to an `AttributeError`.

### Expected behavior

Expected: Successful execution. No output. Worked previously with CUDA 11.3

Retourned:
```
Traceback (most recent call last):
  File "test.py", line 21, in <module>
    model(input_ids, attention_mask=attention_mask)
  File "/home/cdallago/miniconda3/envs/ppih/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/cdallago/miniconda3/envs/ppih/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py", line 1858, in forward
    encoder_outputs = self.encoder(
  File "/home/cdallago/miniconda3/envs/ppih/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/cdallago/miniconda3/envs/ppih/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py", line 1052, in forward
    layer_outputs = layer_module(
  File "/home/cdallago/miniconda3/envs/ppih/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/cdallago/miniconda3/envs/ppih/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py", line 736, in forward
    hidden_states = self.layer[-1](hidden_states)
  File "/home/cdallago/miniconda3/envs/ppih/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/cdallago/miniconda3/envs/ppih/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py", line 333, in forward
    forwarded_states = self.DenseReluDense(forwarded_states)
  File "/home/cdallago/miniconda3/envs/ppih/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/cdallago/miniconda3/envs/ppih/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py", line 292, in forward
    hidden_states = self.wo(hidden_states)
  File "/home/cdallago/miniconda3/envs/ppih/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/cdallago/miniconda3/envs/ppih/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
```

Alternate error when assigning values directly from dictionary object:
```
Traceback (most recent call last):
  File "test.py", line 21, in <module>
    model(input_ids, attention_mask=attention_mask)
  File "/home/cdallago/miniconda3/envs/ppih/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/cdallago/miniconda3/envs/ppih/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py", line 1858, in forward
    encoder_outputs = self.encoder(
  File "/home/cdallago/miniconda3/envs/ppih/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/cdallago/miniconda3/envs/ppih/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py", line 943, in forward
    input_shape = input_ids.size()
AttributeError: 'list' object has no attribute 'size'
```

Suspected: torch incompatibility.

Comments:
Hi @sacdallago 
This is an issue that has been fixed in https://github.com/huggingface/transformers/pull/21281 , if you try to run the script on the `main` branch it should work. `pip install --upgrade pip+https://github.com/huggingface/transformers.git`
Also I see that your inference script might be not optimized, please take a look at `flan-t5` model card on how to run an inference using `T5` models: https://huggingface.co/google/flan-t5-xxl (especially on how you deal with the input)

1. Thanks for the quick fix. It worked, although there was a small typo in the command `pip -> git`, i.e.: ` pip install --upgrade git+https://github.com/huggingface/transformers.git`
2. Thanks for the suggestion, I'll look into it.


Closing for now as the original issue has been fixed