Unable to run models bert/roberta/others w. FP16
Author: jeffra
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.19.2
- Platform: Linux-5.13.0-41-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.5.1
- PyTorch version (GPU?): 1.11.0+cu113 (True)
- Tensorflow version (GPU?): 2.8.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no
```


### Who can help?

@sgugger, @Narsil

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

For several reasons (some performance related) we'd like to be able to run inference on a GPU w. bert-style models in fp16. Unfortunately, I don't believe this mode is currently supported? Unless I am not aware of the right parameter to pass during `pipeline` creation. Below is a code snippet to reproduce the behavior we are seeing.

```python
from transformers import pipeline
pipe = pipeline('fill-mask', model='bert-base-uncased', device=0, framework='pt')

# convert model to fp16
pipe.model.half()

response = pipe('Paris is the [MASK] of France.')
print(response)
```

When running this we see the following stack trace:

```
Traceback (most recent call last):
  File "test.py", line 4, in <module>
    response = pipe('Paris is the [MASK] of France.')
  File "/home/jerasley/ds-env/lib/python3.8/site-packages/transformers/pipelines/fill_mask.py", line 227, in __call__
    outputs = super().__call__(inputs, **kwargs)
  File "/home/jerasley/ds-env/lib/python3.8/site-packages/transformers/pipelines/base.py", line 1026, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
  File "/home/jerasley/ds-env/lib/python3.8/site-packages/transformers/pipelines/base.py", line 1034, in run_single
    outputs = self.postprocess(model_outputs, **postprocess_params)
  File "/home/jerasley/ds-env/lib/python3.8/site-packages/transformers/pipelines/fill_mask.py", line 118, in postprocess
    probs = logits.softmax(dim=-1)
RuntimeError: "softmax_lastdim_kernel_impl" not implemented for 'Half'
```

The core issue in the stacktrace is from the fact that the logits are on the CPU and torch doesn't have a CPU implementation of softmax that works with fp16. I tried moving the model outputs to GPU but then saw several errors related to numpy calls that are not supported on GPU. One workaround (that's maybe not ideal) is if the model outputs are fp16 then we can upcast them to fp32. Would that be an acceptable workaround? If so, I am happy to make a PR that does this.

/cc @RezaYazdaniAminabadi, @mrwyattii, @cli99, @stas00

### Expected behavior

The pipeline should run successfully if the model itself is in fp16 or fp32.



Comments:
We could upcast the outputs to FP32 while transferring them back to CPU since most operations are not implemented in half on CPU. Wdyt @Narsil ?

Seems reasonable, is it no-op for already fp32 ?

We will probably check they are half though since some tensors might contain `int{8,32,64}` which we shouldn't change to float I think, right ?

I agree, I think we'd only want to upcast if a tensor's dtype is fp16. Which would be a no-op if the tensor(s) are already fp32.