VitsTokenizer decode without special tokens produces odd results
Author: xenova
Pull request: no

### System Info

- `transformers` version: 4.35.2
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.0+cu121 (False)
- Tensorflow version (GPU?): 2.15.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.7.5 (cpu)
- Jax version: 0.4.20
- JaxLib version: 0.4.20
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


### Who can help?

@ArthurZucker (tokenizers) @Vaibhavs10 @sanchit-gandhi (audio team)

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```py
>>> from transformers import AutoTokenizer
>>> tokenizer=AutoTokenizer.from_pretrained('facebook/mms-tts-eng')
>>> tokenizer.encode('hello world')
[0, 6, 0, 7, 0, 21, 0, 21, 0, 22, 0, 19, 0, 9, 0, 22, 0, 25, 0, 21, 0, 5, 0]
>>> tokenizer.decode(tokenizer.encode('hello world'), skip_special_tokens=False)
'hello world'
>>> tokenizer.decode(tokenizer.encode('hello world'), skip_special_tokens=True)
'el ol'
>>> tokenizer.decode(tokenizer.encode('abcdefghijklmnopqrstuvwxyz'), skip_special_tokens=True)
'bdfhjmoqsuwy'
```

From the last example, it looks like it's taking the even-positioned elements.

### Expected behavior

`[0, 6, 0, 7, 0, 21, 0, 21, 0, 22, 0, 19, 0, 9, 0, 22, 0, 25, 0, 21, 0, 5, 0]`, for which the tokenized version is:
```
['k', 'h', 'k', 'e', 'k', 'l', 'k', 'l', 'k', 'o', 'k', ' ', 'k', 'w', 'k', 'o', 'k', 'r', 'k', 'l', 'k', 'd', 'k']
```
should be decoded as 'hello world', or something more informative than 'el ol'.

Comments:
I understand users won't typically use the decode method, but it's still quite odd behaviour 😅 

👿 I'm using my joker for this on, it's on the audio team 🤭 

Here's the culprit:
https://github.com/huggingface/transformers/blob/1d7773594754457ed4a79cf6d98bcaabea5bff51/src/transformers/models/vits/tokenization_vits.py#L225-L228

Thanks for reporting @xenova! The VITS MMS tokenizer is very much non-standard in the sense that it uses the same token id as the special pad token as well as a standard vocab token => this is something we probably should have changed when we integrated the model! Is there a case where you'd need to encode and subsequently decode the input text? Happy to take a look at a fix if so! Otherwise, we can add a disclaimer to the docs saying that `tokenizer.decode(tokenizer.encode(input_text)) != input_text ` due to this behaviour

Copying a related discussion from Slack:

VB
  [23 days ago](https://huggingface.slack.com/archives/C02G13FEMDH/p1703055571159369?thread_ts=1703042642.253789&cid=C02G13FEMDH)
Yes, it is the pad token: https://huggingface.co/facebook/mms-tts-eng/blob/main/tokenizer_config.json#L8
AFAIK, k represents the character boundary (to seperate one character from the other). Since the text is split and converted to phonemic representation later, this helps segregate it. each character has a seperate phonemic representation.
each phonemic representation creates it's respective mel-spec based on the duration prediction.
k is later on ignored in during attention as the mask is set to 1 i.e. do not attend to it for it.
https://huggingface.co/facebook/mms-tts-eng/blob/c71de0fe7204c83f1c10820a7d696d0b450048ba/vocab.json#L23
I think the reason for choosing k  in case of MMS was to make it be able to generalise across the multitudes of language they trained on.


Joshua Lochner
  [23 days ago](https://huggingface.slack.com/archives/C02G13FEMDH/p1703074142921659?thread_ts=1703042642.253789&cid=C02G13FEMDH)
Oh wow that’s so interesting :joy: Thanks!


Joshua Lochner
  [23 days ago](https://huggingface.slack.com/archives/C02G13FEMDH/p1703074186957259?thread_ts=1703042642.253789&cid=C02G13FEMDH)
I also checked some of the other languages, and indeed they also set a pad token to be one of those characters the vocabulary.


Sanchit
  [23 days ago](https://huggingface.slack.com/archives/C02G13FEMDH/p1703075297154269?thread_ts=1703042642.253789&cid=C02G13FEMDH)
Yes it’s a bit messy, but they essentially just picked one of the existing characters in the MMS tokenizer to act as the padding token (in this case k), and then mask it during the forward pass with the attention mask
In the MMS integration to transformers I used the same behaviour they did in the original repo (use k as the padding token). Probably what would be more elegant is:
1. Adding a new padding token to the MMS vocab (e.g. <pad>)
2. Re-sizing the embedding layer to account for this new padding token with randomly initialised weights
3. Mask out the embeddings that come from the padding token in the forward pass (as is done already)
This would break the existing MMS weights though, so unfortunately can’t be done now, but would have been a better option in retrospect (edited) 


Sanchit
  [23 days ago](https://huggingface.slack.com/archives/C02G13FEMDH/p1703075337870219?thread_ts=1703042642.253789&cid=C02G13FEMDH)
(The solution I explained above is how they do it in VITS with a dedicated <pad> token)


Amy
  [23 days ago](https://huggingface.slack.com/archives/C02G13FEMDH/p1703079737456419?thread_ts=1703042642.253789&cid=C02G13FEMDH)
There’s a related discussion here: https://github.com/huggingface/transformers/pull/24085#discussion_r1241240099

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.