[WIP] [Whisper] Fix generate and tokenizer behavior with added tokens
Author: eustlb
Pull request: yes

# What does this PR do?

Fixes #33082 #35330.

### Description of the issue

Transformers' `PretTrainedTokenizer` [`_add_tokens`](https://github.com/huggingface/transformers/blob/ce62a41880b5b70a304d068eb58f55894a5a7af8/src/transformers/tokenization_utils.py#L510) add tokens at the end of the vocabulary. Likewise, `PreTrainedModel`'s [`_get_resized_embeddings`](https://github.com/huggingface/transformers/blob/ce62a41880b5b70a304d068eb58f55894a5a7af8/src/transformers/modeling_utils.py#L2080) will add newly initialized tokens at the end. This behavior is not compatible with Whisper's timestamp token identification. Indeed, official implementation as well as Transformer's one identifies timestamp tokens as the ones that have an id > `timestamp_begin`. For this reason, the newly added tokens are falsely considered timestamps and this poses decoding issues. 

###  Implementation possibilities

Two possibilities here: 
1. change Whisper's decoding logic using a timestamp_end
2. overwrite  [`_add_tokens`](https://github.com/huggingface/transformers/blob/ce62a41880b5b70a304d068eb58f55894a5a7af8/src/transformers/tokenization_utils.py#L510)  and [`_get_resized_embeddings`](https://github.com/huggingface/transformers/blob/ce62a41880b5b70a304d068eb58f55894a5a7af8/src/transformers/modeling_utils.py#L2080) in order to add new tokens before the first timestamp token (and not at the end as current implementation)

I chose option 1 to avoid overwriting the default methods of the Transformers library, focusing instead on modifying the Whisper-specific generation method and tokenizer logic.

## Implementation decision (see below discussion for details)

In the updated implementation, Whisper will now use both `timestamp_begin` and `timestamp_end`, instead of just `timestamp_begin` as in OpenAI’s original version.

**→  Justification:**

**what is the best for the users (least amount of work for them)**

We have two scenarios:

1.	Adding text tokens.
2.	Adding timestamp tokens.

These cases overlap, as both types of tokens must be added via the add_tokens method. For the system to work seamlessly:

- Text tokens should be inserted before the first timestamp token (Case 1).
- Timestamp tokens should be added at the end of the vocabulary (Case 2).

To handle Case 1, modifying the `add_tokens` method would also require changes to the `resize_token_embedding`. From a simplicity perspective, deviating from the standard implementation of these methods complicates things unnecessarily.

Since Case 1 is more common, we should prioritize simplifying it for the user. Hardcoding a default number of timestamp tokens achieves this, and the `number_timestamp_tokens` config option easily accommodates Case 2 for the few users who need it.

As for the tokenizer, the regex pattern currently used to detect timestamps prevents the use of custom timestamp tokens (meaning with a custom syntax not matching the pattern). However, since this is a rare use case and the current implementation already depends on regex, it’s best to keep things simple and do it this way.

## Who can review?

@ylacombe 

# TODO 

- [x] discuss different implementations and choose 
- [ ] add tests before merging

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33512). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

**→  Let's wrap up a bit and justify choices further:** 

In the updated implementation, Whisper will now use both `timestamp_begin` and `timestamp_end`, instead of just `timestamp_begin` as in OpenAI’s original version.

**→  Justification:**

**what is the best for the users (least amount of work for them)**

We have two scenarios:

1.	Adding text tokens.
2.	Adding timestamp tokens.

These cases overlap, as both types of tokens must be added via the add_tokens method. For the system to work seamlessly:

- Text tokens should be inserted before the first timestamp token (Case 1).
- Timestamp tokens should be added at the end of the vocabulary (Case 2).

To handle Case 1, modifying the `add_tokens` method would also require changes to the `resize_token_embedding`. From a simplicity perspective, deviating from the standard implementation of these methods complicates things unnecessarily.

Since Case 1 is more common, we should prioritize simplifying it for the user. Hardcoding a default number of timestamp tokens achieves this, and the `number_timestamp_tokens` config option easily accommodates Case 2 for the few users who need it.

As for the tokenizer, the regex pattern currently used to detect timestamps prevents the use of custom timestamp tokens (meaning with a custom syntax not matching the pattern). However, since this is a rare use case and the current implementation already depends on regex, it’s best to keep things simple and do it this way.