tokenizer decode decode with timestamp fails for extended vocabulary
Author: bnestor
Pull request: no
Labels: name: bug

### System Info

python=3.10.13
transformers==4.44.1
torch==2.1.2

### Who can help?

@sanchit-gandhi @ylacombe @eustlb @arthurz

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Decoding with timestamps produces unexpected results when the vocabulary is extended

```
>>> from transformers import WhisperTokenizer, AddedToken
>>> tokenizer = WhisperTokenizer.from_pretrained('openai/whisper-base', language="English", task="transcribe", predict_timestamps=True)
>>> extended_vocab = ['newword1']
>>> extended_vocab = [AddedToken(t, single_word=True, lstrip=True) for t in extended_vocab]
>>> tokenizer.add_tokens(extended_vocab)
1
>>> print(len(tokenizer))
51866
>>> print(tokenizer.convert_ids_to_tokens(51865))
newword1
>>> tokens = tokenizer('<|0.00|> newword1 <|0.22|>').input_ids
>>> tokens
[50258, 50259, 50359, 50364, 51865, 220, 50375, 50257]
>>> tokenizer.decode(tokens, skip_special_tokens=True)
'newword1 '
>>> tokenizer.decode(tokens, skip_special_tokens=False)
'


newword1 
'
>>> tokenizer.decode(tokens, skip_special_tokens=False, decode_with_timestamps=True)
'


<|0.00|><|30.02|> <|30.24|>
'
>>> tokens = tokenizer('<|0.00|> word <|0.22|>').input_ids # something in the vocabulary
>>> tokenizer.decode(tokens, skip_special_tokens=True)
' word '
>>> tokenizer.decode(tokens, skip_special_tokens=False)
'


 word 
'
>>> tokenizer.decode(tokens, skip_special_tokens=False, decode_with_timestamps=True)
'


<|0.00|> word <|0.22|>
'
```

The problem arises in [https://github.com/huggingface/transformers/blob/9613933b022ddbf085e2c593ed4ceea4c734179a/src/transformers/models/whisper/tokenization_whisper.py#L546]( https://github.com/huggingface/transformers/blob/9613933b022ddbf085e2c593ed4ceea4c734179a/src/transformers/models/whisper/tokenization_whisper.py#L546)

see issue [20225](https://github.com/huggingface/transformers/issues/20225)


### Expected behavior

I would expect the timestamps to remain consistent from tokenizing and decoding.

```
>>> tokens = tokenizer('<|0.00|> newword1 <|0.22|>').input_ids
>>> tokenizer.decode(tokens, skip_special_tokens=False, decode_with_timestamps=True)
'


<|0.00|> newword1<|0.22|>
'
```

Comments:
Hey @bnestor, thanks a lot for raising this issue.
Indeed the problem arises where you've spot it, it's linked to https://github.com/huggingface/transformers/issues/33082 and the PR  I already did to fix it https://github.com/huggingface/transformers/pull/33512. It had to write tests for it before merging but we then underwent another bug-fixing effort (https://github.com/huggingface/transformers/pull/34535, https://github.com/huggingface/transformers/pull/34537, https://github.com/huggingface/transformers/pull/34111) making the PR to stall, sorry for the delay! Nevertheless, it's next point on the whisper bug-fixing roadmap so should be solved quickly 🤗

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

> Hey [@bnestor](https://github.com/bnestor), thanks a lot for raising this issue. Indeed the problem arises where you've spot it, it's linked to [#33082](https://github.com/huggingface/transformers/issues/33082) and the PR I already did to fix it [#33512](https://github.com/huggingface/transformers/pull/33512). It had to write tests for it before merging but we then underwent another bug-fixing effort ([#34535](https://github.com/huggingface/transformers/pull/34535), [#34537](https://github.com/huggingface/transformers/pull/34537), [#34111](https://github.com/huggingface/transformers/pull/34111)) making the PR to stall, sorry for the delay! Nevertheless, it's next point on the whisper bug-fixing roadmap so should be solved quickly 🤗

Hi @eustlb, wondering whether the PR for fixing this issue is still at a stall or dropped entirely. What would be the quick workaround for using vocab expansion and timestamp decoding (if the PR isn't merged yet)?

From what I read, your proposed solution should be the best thing for now in doing non-timestamp vocab expansion while preserving text and timestamp decoding behavior, because in my usecase of Special Token Expansion, the issue with the current one is the timestamp can't be decoded due to the logic of checking timestamp start IDs based on last special token ids (which is changed to the last due to vocab expansion).

Hi @sabilmakbar, ahhh long time no see this issue! A lot has happened in the meantime, and merging my solution is still not a priority. That said, I’ll try to find a time window to do it, just to ease my conscience 😅. In the meantime, feel free to use what I did in #33512.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.