Incorrect whisper word-level timestamps
Author: xenova
Pull request: no

### System Info

- `transformers` version: 4.32.0.dev0 (main)
- Platform: Linux-5.15.0-1041-azure-x86_64-with-glibc2.31
- Python version: 3.10.8
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.0.1+cu117 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


### Who can help?

@sanchit-gandhi

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction


Running
```python
from transformers import pipeline
pipe = pipeline("automatic-speech-recognition", "openai/whisper-base")
url = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/japanese-audio.wav'
output = pipe(url, return_timestamps="word", chunk_length_s=30, generate_kwargs={'language': 'japanese'})
print(output)
```

Produces
```
{'text': '森長の美味しい牛乳は濃い青いように牛乳ビーンを足らった絶対にのパック牛乳である', 'chunks': [{'text': '森', 'timestamp': (29.98, 29.98)}, {'text': '長', 'timestamp': (29.98, 29.98)}, {'text': 'の', 'timestamp': (29.98, 29.98)}, {'text': '美味', 'timestamp': (29.98, 29.98)}, {'text': 'しい', 'timestamp': (29.98, 29.98)}, {'text': '牛', 'timestamp': (29.98, 29.98)}, {'text': '乳', 'timestamp': (29.98, 29.98)}, {'text': 'は', 'timestamp': (29.98, 29.98)}, {'text': '濃', 'timestamp': (29.98, 29.98)}, {'text': 'い', 'timestamp': (29.98, 29.98)}, {'text': '青', 'timestamp': (29.98, 29.98)}, {'text': 'い', 'timestamp': (29.98, 29.98)}, {'text': 'ように', 'timestamp': (29.98, 29.98)}, {'text': '牛', 'timestamp': (29.98, 29.98)}, {'text': '乳', 'timestamp': (29.98, 29.98)}, {'text': 'ビ', 'timestamp': (29.98, 29.98)}, {'text': 'ーン', 'timestamp': (29.98, 29.98)}, {'text': 'を', 'timestamp': (29.98, 29.98)}, {'text': '足', 'timestamp': (29.98, 29.98)}, {'text': 'ら', 'timestamp': (29.98, 29.98)}, {'text': 'った', 'timestamp': (29.98, 29.98)}, {'text': '絶', 'timestamp': (29.98, 29.98)}, {'text': '対', 'timestamp': (29.98, 29.98)}, {'text': 'に', 'timestamp': (29.98, 29.98)}, {'text': 'の', 'timestamp': (29.98, 29.98)}, {'text': 'パ', 'timestamp': (29.98, 29.98)}, {'text': 'ック', 'timestamp': (29.98, 29.98)}, {'text': '牛', 'timestamp': (29.98, 29.98)}, {'text': '乳', 'timestamp': (29.98, 29.98)}, {'text': 'で', 'timestamp': (29.98, 29.98)}, {'text': 'ある', 'timestamp': (29.98, 29.98)}]}
```

### Expected behavior

Should not have all timestamps = 29.98. It should be noted that `29.98 = 30 - 0.02 = chunk_length_s - time_precision`, which I don't think is a coincidence.

Output should better-match the output from openai's whisper library:

 ```python
import whisper
model = whisper.load_model("base")
url = "https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/japanese-audio.wav"
result = model.transcribe(url, word_timestamps=True)
print(result)
```

```
{'text': '森長の美味しい牛乳は濃い青いように牛乳ビーンを足らった絶対にのパック牛乳である', 'segments': [{'id': 0, 'seek': 0, 'start': 0.28, 'end': 6.06, 'text': '森長の美味しい牛乳は濃い青いように牛乳ビーンを足らった絶対にのパック牛乳である', 'tokens': [50364, 16407, 106, 15353, 2972, 45511, 17121, 40003, 2930, 111, 3065, 26373, 225, 1764, 37462, 1764, 34483, 40003, 2930, 111, 39454, 44632, 5998, 37236, 5154, 10102, 6948, 114, 35252, 4108, 2972, 23268, 28551, 40003, 2930, 111, 2474, 24719, 50776], 'temperature': 0.0, 'avg_logprob': -0.5187992572784423, 'compression_ratio': 1.1142857142857143, 'no_speech_prob': 0.06956221163272858, 'words': [{'word': '森', 'start': 0.28, 'end': 0.64, 'probability': 0.6905971467494965}, {'word': '長', 'start': 0.64, 'end': 0.82, 'probability': 0.25675880908966064}, {'word': 'の', 'start': 0.82, 'end': 1.04, 'probability': 0.9857621192932129}, {'word': '美味', 'start': 1.04, 'end': 1.18, 'probability': 0.44574692845344543}, {'word': 'しい', 'start': 1.18, 'end': 1.48, 'probability': 0.9633633494377136}, {'word': '牛', 'start': 1.48, 'end': 1.68, 'probability': 0.9765644073486328}, {'word': '乳', 'start': 1.68, 'end': 1.94, 'probability': 0.8430313766002655}, {'word': 'は', 'start': 1.94, 'end': 2.12, 'probability': 0.9037646651268005}, {'word': '濃', 'start': 2.12, 'end': 2.34, 'probability': 0.7717265486717224}, {'word': 'い', 'start': 2.34, 'end': 2.46, 'probability': 0.9223815202713013}, {'word': '青', 'start': 2.46, 'end': 2.64, 'probability': 0.7740068435668945}, {'word': 'い', 'start': 2.64, 'end': 2.76, 'probability': 0.861002504825592}, {'word': 'ように', 'start': 2.76, 'end': 2.92, 'probability': 0.10338784009218216}, {'word': '牛', 'start': 2.92, 'end': 3.14, 'probability': 0.8816423416137695}, {'word': '乳', 'start': 3.14, 'end': 3.44, 'probability': 0.9983818531036377}, {'word': 'ビ', 'start': 3.44, 'end': 3.58, 'probability': 0.3014017343521118}, {'word': 'ーン', 'start': 3.58, 'end': 3.7, 'probability': 0.7804636359214783}, {'word': 'を', 'start': 3.7, 'end': 3.8, 'probability': 0.9925546050071716}, {'word': '足', 'start': 3.8, 'end': 3.98, 'probability': 0.26062318682670593}, {'word': 'ら', 'start': 3.98, 'end': 4.1, 'probability': 0.7511312365531921}, {'word': 'った', 'start': 4.1, 'end': 4.3, 'probability': 0.7001805901527405}, {'word': '絶', 'start': 4.3, 'end': 4.52, 'probability': 0.5361797697842121}, {'word': '対', 'start': 4.52, 'end': 4.68, 'probability': 0.27607205510139465}, {'word': 'に', 'start': 4.68, 'end': 4.78, 'probability': 0.40600043535232544}, {'word': 'の', 'start': 4.78, 'end': 4.94, 'probability': 0.8449609875679016}, {'word': 'パ', 'start': 4.94, 'end': 5.1, 'probability': 0.43969523906707764}, {'word': 'ック', 'start': 5.1, 'end': 5.22, 'probability': 0.916713297367096}, {'word': '牛', 'start': 5.22, 'end': 5.42, 'probability': 0.9680136442184448}, {'word': '乳', 'start': 5.42, 'end': 5.7, 'probability': 0.9995707273483276}, {'word': 'で', 'start': 5.7, 'end': 5.84, 'probability': 0.9711904525756836}, {'word': 'ある', 'start': 5.84, 'end': 6.06, 'probability': 0.9810820817947388}]}], 'language': 'ja'}
```

---

Fortunately, the output from the models are identical, so there's no issue with `generate`.

```
# transformers: 森長の美味しい牛乳は濃い青いように牛乳ビーンを足らった絶対にのパック牛乳である
# openai: 森長の美味しい牛乳は濃い青いように牛乳ビーンを足らった絶対にのパック牛乳である
```

Comments: