Run `AutomaticSpeechRecognitionPipeline` with FP16
Author: bofenghuang
Pull request: no

### Feature request

Hi @Narsil,

I would like to run inference with `AutomaticSpeechRecognitionPipeline` in FP16 using some large models (e,g, whisper). But I don't believe it's supported in current version (please correct me if I'm wrong here).

### Reproduction

Below is a code snippet to reproduce the behavior.

```python
import torch
from transformers import pipeline

pipe = pipeline(model="openai/whisper-base", device=0, torch_dtype=torch.float16)
pipe("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/1.flac")
```

When running this we see the following stack trace:

```
RuntimeError                              Traceback (most recent call last)
Cell In[1], line 5
      2 from transformers import pipeline
      4 pipe = pipeline(model="openai/whisper-base", device=0, torch_dtype=torch.float16)
----> 5 pipe("https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/1.flac")

File ~/transformers/src/transformers/pipelines/automatic_speech_recognition.py:232, in AutomaticSpeechRecognitionPipeline.__call__(self, inputs, **kwargs)
    191 def __call__(
    192     self,
    193     inputs: Union[np.ndarray, bytes, str],
    194     **kwargs,
    195 ):
    196     """
    197     Transcribe the audio sequence(s) given as inputs to text. See the [`AutomaticSpeechRecognitionPipeline`]
    198     documentation for more information.
   (...)
    230                     `"".join(chunk["text"] for chunk in output["chunks"])`.
    231     """
--> 232     return super().__call__(inputs, **kwargs)

File ~/transformers/src/transformers/pipelines/base.py:1074, in Pipeline.__call__(self, inputs, num_workers, batch_size, *args, **kwargs)
   1072     return self.iterate(inputs, preprocess_params, forward_params, postprocess_params)
   1073 else:
-> 1074     return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)

File ~/transformers/src/transformers/pipelines/base.py:1096, in ChunkPipeline.run_single(self, inputs, preprocess_params, forward_params, postprocess_params)
   1094 all_outputs = []
   1095 for model_inputs in self.preprocess(inputs, **preprocess_params):
-> 1096     model_outputs = self.forward(model_inputs, **forward_params)
   1097     all_outputs.append(model_outputs)
   1098 outputs = self.postprocess(all_outputs, **postprocess_params)

File ~/transformers/src/transformers/pipelines/base.py:990, in Pipeline.forward(self, model_inputs, **forward_params)
    988     with inference_context():
    989         model_inputs = self._ensure_tensor_on_device(model_inputs, device=self.device)
--> 990         model_outputs = self._forward(model_inputs, **forward_params)
    991         model_outputs = self._ensure_tensor_on_device(model_outputs, device=torch.device("cpu"))
    992 else:

File ~/transformers/src/transformers/pipelines/automatic_speech_recognition.py:370, in AutomaticSpeechRecognitionPipeline._forward(self, model_inputs)
    364     # we need to pass `processed.get("attention_mask")` here since audio encoder
    365     # attention mask  length is different from expected text decoder `encoder_attention_mask` length
    366     # `generate` magic to create the mask automatically won't work, we basically need to help
    367     # it here.
    368     attention_mask = model_inputs.pop("attention_mask", None)
    369     tokens = self.model.generate(
--> 370         encoder_outputs=encoder(inputs, attention_mask=attention_mask),
    371         attention_mask=attention_mask,
    372     )
    374     out = {"tokens": tokens}
    376 else:

File ~/anaconda3/envs/asr/lib/python3.8/site-packages/torch/nn/modules/module.py:1190, in Module._call_impl(self, *input, **kwargs)
   1186 # If we don't have any hooks, we want to skip the rest of the logic in
   1187 # this function, and just call forward.
   1188 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1189         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1190     return forward_call(*input, **kwargs)
   1191 # Do not call functions when jit is used
   1192 full_backward_hooks, non_full_backward_hooks = [], []

File ~/transformers/src/transformers/models/whisper/modeling_whisper.py:654, in WhisperEncoder.forward(self, input_features, attention_mask, head_mask, output_attentions, output_hidden_states, return_dict)
    650 output_hidden_states = (
    651     output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
    652 )
    653 return_dict = return_dict if return_dict is not None else self.config.use_return_dict
--> 654 inputs_embeds = nn.functional.gelu(self.conv1(input_features))
    655 inputs_embeds = nn.functional.gelu(self.conv2(inputs_embeds))
    657 inputs_embeds = inputs_embeds.permute(0, 2, 1)

File ~/anaconda3/envs/asr/lib/python3.8/site-packages/torch/nn/modules/module.py:1190, in Module._call_impl(self, *input, **kwargs)
   1186 # If we don't have any hooks, we want to skip the rest of the logic in
   1187 # this function, and just call forward.
   1188 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1189         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1190     return forward_call(*input, **kwargs)
   1191 # Do not call functions when jit is used
   1192 full_backward_hooks, non_full_backward_hooks = [], []

File ~/anaconda3/envs/asr/lib/python3.8/site-packages/torch/nn/modules/conv.py:313, in Conv1d.forward(self, input)
    312 def forward(self, input: Tensor) -> Tensor:
--> 313     return self._conv_forward(input, self.weight, self.bias)

File ~/anaconda3/envs/asr/lib/python3.8/site-packages/torch/nn/modules/conv.py:309, in Conv1d._conv_forward(self, input, weight, bias)
    305 if self.padding_mode != 'zeros':
    306     return F.conv1d(F.pad(input, self._reversed_padding_repeated_twice, mode=self.padding_mode),
    307                     weight, bias, self.stride,
    308                     _single(0), self.dilation, self.groups)
--> 309 return F.conv1d(input, weight, bias, self.stride,
    310                 self.padding, self.dilation, self.groups)

RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.cuda.HalfTensor) should be the same
```

### System Info

```
- `transformers` version: 4.26.0.dev0
```

### Motivation

To accelerate inference and take less memory when using the pipeline with large models

### Your contribution

Right now I just force the casting to fp16 after calling `feature_extractor` to make the pipeline run for whisper models (I'm not yet using chunked inference).

https://github.com/huggingface/transformers/blob/3090e708577e2d0145ab81d0e2362e3235aebbd9/src/transformers/pipelines/automatic_speech_recognition.py#L338-L340

```python
processed["input_features"] = processed["input_features"].to(self.model.config.torch_dtype)
```

But I understand should better wrap it in another function which considers input names for different models. Willing to make a PR if you can guide me here :)

Comments:
Good catch ! Thanks for the tip.

I started considering a few options into how this could be done, and I ended up doing the PR, please review if you can (sorry for doing it, I usually like when contributors can do it, but as I wasn't sure of what exactly should be done and I was exploring, I ended up doing it :D )

https://github.com/huggingface/transformers/pull/20864

No problem! Thanks for the explication in the PR! Just leave some comments