WhisperProcessor doesn't copy output tensor to CPU for `decode(output_offsets=True)`
Author: rklasen
Pull request: no

### System Info

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.36.1
- Platform: Linux-6.6.7-arch1-1-x86_64-with-glibc2.38
- Python version: 3.11.6
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: YES (this is the cause of the bug)
- Using distributed or parallel set-up in script?: no, one local GPU 

### Who can help?

@sanchit-gandhi 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I'm creating a WhisperProcessor, copy input and model to GPU and run `generate()` with `return_timestamps=True`.  

```python
processor = WhisperProcessor.from_pretrained("openai/whisper-large-v2")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v2")

model = model.to("cuda")
input_features = input_features.to("cuda")

with torch.backends.cuda.sdp_kernel(enable_flash=True, enable_math=True, enable_mem_efficient=True):
    output_with_prompt = model.generate(input_features, return_timestamps=True)

result = processor.decode(output_with_prompt[0], skip_special_tokens=True, decode_with_timestamps=False, output_offsets=True)
print(result)
```

Causes the error:

```
File [/media/DataStore02-12TB/codeRepos/myFasterWhisperTest/.venv/lib/python3.11/site-packages/torch/_tensor.py:1030](https://file+.vscode-resource.vscode-cdn.net/media/DataStore02-12TB/codeRepos/myFasterWhisperTest/.venv/lib/python3.11/site-packages/torch/_tensor.py:1030), in Tensor.__array__(self, dtype)
   [1028](https://file+.vscode-resource.vscode-cdn.net/media/DataStore02-12TB/codeRepos/myFasterWhisperTest/.venv/lib/python3.11/site-packages/torch/_tensor.py:1028)     return handle_torch_function(Tensor.__array__, (self,), self, dtype=dtype)
   [1029](https://file+.vscode-resource.vscode-cdn.net/media/DataStore02-12TB/codeRepos/myFasterWhisperTest/.venv/lib/python3.11/site-packages/torch/_tensor.py:1029) if dtype is None:
-> [1030](https://file+.vscode-resource.vscode-cdn.net/media/DataStore02-12TB/codeRepos/myFasterWhisperTest/.venv/lib/python3.11/site-packages/torch/_tensor.py:1030)     return self.numpy()
   [1031](https://file+.vscode-resource.vscode-cdn.net/media/DataStore02-12TB/codeRepos/myFasterWhisperTest/.venv/lib/python3.11/site-packages/torch/_tensor.py:1031) else:
   [1032](https://file+.vscode-resource.vscode-cdn.net/media/DataStore02-12TB/codeRepos/myFasterWhisperTest/.venv/lib/python3.11/site-packages/torch/_tensor.py:1032)     return self.numpy().astype(dtype, copy=False)

TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.
```

Copying the output tensor to the CPU before fixes that:

```
output_with_prompt2=output_with_prompt.cpu()

# timesatmps must be on to skip the initial prompt
result = processor.decode(output_with_prompt2[0], skip_special_tokens=True, decode_with_timestamps=False, output_offsets=True)
print(result)
``` 

However, that **only** happens when `output_offsets=True` is enabled. When the flag is disabled, the decode works fine on the GPU (but we don't get the timestamps).

I'm also seeing that the decode function is called about 11 times for one `processor.decode`, is that by design?

### Expected behavior

Automatically copy the tensor to CPU for the decode.  

Comments: