Wrong pre-trained Whisper's BOS token?
Author: tonywu71
Pull request: no

### System Info

- `transformers` version: 4.30.2
- Platform: macOS-13.4-arm64-arm-64bit
- Python version: 3.10.11
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@sanchit

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import WhisperTokenizer
WhisperTokenizer.from_pretrained("openai/whisper-tiny").bos_token
>> '
'
```

### Expected behavior

Dear Gandhi,

From the [documentation](https://huggingface.co/docs/transformers/model_doc/whisper#transformers.WhisperTokenizer) and from what I expect from the Whisper tokenizer, `processor.tokenizer.bos_token` should be equal to `"
"` when using one of the official vanilla Whisper model. Currently, it is equal to `"
"`. Is it an intended behavior? What do you think?

On a different note, there is another weird behavior when encoding/decoding:

```python
tokenizer.encode(["
"])
>> [50258, 50363, 50258, 50257]
processor.tokenizer.decode([50258, 50363, 50258, 50257])
>> '



'
```

while I was expecting the last line to return `'
'` only.

Yours sincerely,
Tony

Comments:
Hi!

The first issue seems to be a feature of the whisper model. It has `
` as token text for `bos`, `eos`, `pad` and `unk`. I see there are no dedicated tokens for `unk` and `pad`, so I think this is a feature of the model, and not a bug. If you look at the [original code](https://github.com/openai/whisper/blob/main/whisper/tokenizer.py), you can see that there is no dedicated token for `eos`, `bos`, `pad` or `unk`. This seems to indicate that these tokens are simply not used by the model.

The second issue is due to `add_special_tokens` being set to `True` by default. So this is not unexpected behavior.

```python
tokenizer.encode(["
"], add_special_tokens=False)
>>> [50258]
tokenizer.decode([50258])
>>> '
'
```




cc @ArthurZucker 

Hey, not entirely sure which part of the documentation you are referring to, but this is expected. The `bos_token` is not used to start a transcript. More details [here](https://huggingface.co/openai/whisper-base) about the starting tokens, and why we don't use this `bos`.

> Hi!
> 
> The first issue seems to be a feature of the whisper model. It has `
` as token text for `bos`, `eos`, `pad` and `unk`. I see there are no dedicated tokens for `unk` and `pad`, so I think this is a feature of the model, and not a bug. If you look at the [original code](https://github.com/openai/whisper/blob/main/whisper/tokenizer.py), you can see that there is no dedicated token for `eos`, `bos`, `pad` or `unk`. This seems to indicate that these tokens are simply not used by the model.
> 
> The second issue is due to `add_special_tokens` being set to `True` by default. So this is not unexpected behavior.
> 
> ```python
> tokenizer.encode(["
"], add_special_tokens=False)
> >>> [50258]
> tokenizer.decode([50258])
> >>> '
'
> ```

Thanks, makes sense!

> Hey, not entirely sure which part of the documentation you are referring to, but this is expected. The `bos_token` is not used to start a transcript. More details [here](https://huggingface.co/openai/whisper-base) about the starting tokens, and why we don't use this `bos`.

According to this [part](https://huggingface.co/docs/transformers/model_doc/whisper#transformers.WhisperTokenizer.bos_token), we have:

> bos_token (str, optional, defaults to "
") — The beginning of sequence token.

Which in my opinion is a bit confusing.

But I do understand your point and how I should handle the `
` now. Thanks for the help!

I'll update the documentation to make it less confusing. The token used to store the ` "
"` token is `decoder_start_token_id`. The `bos_token` is  pretty much unused, which is why it was set to the same as `eos_token`.