[Whisper] Fix word-level timestamps with bs>1 or num_beams>1
Author: ylacombe
Pull request: yes

# What does this PR do?

Supersedes #26699

This PR fixes two issues related to Whisper:
1. Wrong DTW matrix computation when computing word-level timestamps with beam search (issues #27362 and #28007)
2. Bug when computing world-levels timestamps with bs>1 using the pipeline (issue #27446 and PR #26699)

The first issue happens because the DTW matrix is derived from the cross attentions. The latter is of size `beam_search*num_return_sequences*batch_size`,  but it should be of size `num_return_sequences*batch_size` , so we need to keep track of the beam indices.

The second issue happens because when batching with the pipeline, `stride` is passed as a list of tuple (one per sample) instead of a single tuple.
When there are multiple strides passed to `_extract_token_timestamps`, we can't compute the DTW matrix in parallel.
It is treated in two cases:

1. If same stride for each sample, compute DTW weights in parallel
2. If not the same stride (i.e end of audio file) compute them sequentially

The loss of parallelism is not so dramatic, since in all cases the DTW algorithm is performed sequentially.



Fixes #27362, #28007, #27446 

cc @sanchit-gandhi, @amyeroberts 



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28114). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Merging now, once the checks are green, as it is a blocking feature, all tests are passing and @gante is off for a few days!

Hi @ylacombe 
Thanks for the fix! I do still face an issue I think is relevant here so I figured I'll post it.

When I'm trying to use the whisper model with batch size > 1 (for my example: 2) and word level timestamps, I get this error in the DTW method:

```
...
File "/transformers/pipelines/base.py", line 926, in device_placement
    yield
File "/transformers/pipelines/base.py", line 1046, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
File "/transformers/pipelines/automatic_speech_recognition.py", line 572, in _forward
    tokens = self.model.generate(
File "/transformers/models/whisper/modeling_whisper.py", line 2273, in generate
    outputs["token_timestamps"] = self._extract_token_timestamps(
File "/transformers/models/whisper/modeling_whisper.py", line 2624, in _extract_token_timestamps
    text_indices, time_indices = _dynamic_time_warping(-matrix.double().cpu().numpy())
File "/transformers/models/whisper/modeling_whisper.py", line 258, in _dynamic_time_warping
    output_length, input_length = matrix.shape
ValueError: too many values to unpack (expected 2)
```

The `matrix` object's shape is `(6, 72, 1500)` and indeed it is expecting a shape of 2 so it fails. Do you have any idea what happened or how can I overcome it?

Thanks!

### Update:
Perhaps the fix might be here:

line 2611 in modeling_whiaper.py:
```
if num_frames is not None and isinstance(num_frames, (tuple, list)):
```
should be:
```
if num_frames is not None and isinstance(num_frames, (tuple, list, np.ndarray)):
```
as from what I saw it is a numpy array.

### Update 2:
That seems to have fixed it. I'll open a PR and if you can, review it :)
https://github.com/huggingface/transformers/pull/28226