AlbertTokenizer doesn't decode special tokens properly
Author: NielsRogge
Pull request: no
Labels: name: Good First Issue

Related to #5142, `AlbertTokenizer` (which uses SentencePiece) doesn't decode special tokens (like [CLS], [MASK]) properly. This issue was discovered when adding the Nystromformer model (#14659), which uses this tokenizer.

To reproduce (Transformers v4.15 or below):

```
!pip install -q transformers sentencepiece

from transformers import AlbertTokenizer

tokenizer = AlbertTokenizer.from_pretrained("albert-base-v1")

text = "hello world"
encoding = tokenizer(text)

for id in encoding.input_ids:
  print(id, tokenizer.decode([id]))
```
This prints:
```
2 
10975 hello
126 world
3 
```
As can be seen, the special tokens are added ([CLS] with ID=2 and [SEP] with id=3), but they are decoded to an empty string. This is because the `convert_tokens_to_string` [method](https://github.com/huggingface/transformers/blob/e68c3756fea7c811d02b8470539ae17ec3ec0e71/src/transformers/models/albert/tokenization_albert.py#L252) of `AlbertTokenizer` uses the decode method of Google's SentencePiece library, but this doesn't take into account special tokens. 

The issue does not occur with the fast tokenizer:
```
from transformers import AlbertTokenizerFast

tokenizer = AlbertTokenizerFast.from_pretrained("albert-base-v1")

text = "hello world"
encoding = tokenizer(text)

for id in encoding.input_ids:
  print(id, tokenizer.decode([id]))
```
Which prints:
```
2 [CLS]
10975 hello
126 world
3 [SEP]
```
A similar issue happened for T5, and this was fixed in #8435. 

Comments:
Update: there are many more SentencePiece tokenizers that may require an update, see #11716

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hello Niels! 👋 

I started taking a look at this issue, and I do think that @patrickvonplaten's method would work in this instance as well, but as you have warned, this problem does indeed creep in at least 8 other SentencePiece-based tokenizers, not all of which have a `*Fast` flavor in order to mimic the test in the T5 case.

First, I imported all mentioned tokenizers in #11716 except `XLMRobertaTokenizer` (which I couldn't find a suitable checkpoint for) like so:

```python
albert_tokenizer = AlbertTokenizer.from_pretrained("albert-base-v1")
barthez_tokenizer = BarthezTokenizer.from_pretrained("moussaKam/barthez")
bert_generation_tokenizer = BertGenerationTokenizer.from_pretrained("google/roberta2roberta_L-24_discofuse")
bigbird_tokenizer = BigBirdTokenizer.from_pretrained("google/bigbird-roberta-base")
camembert_tokenizer = CamembertTokenizer.from_pretrained("camembert-base")
debertav2_tokenizer = DebertaV2Tokenizer.from_pretrained("microsoft/deberta-v2-xlarge")
m2m100_tokenizer = M2M100Tokenizer.from_pretrained("facebook/m2m100_418M")
marian_tokenizer = MarianTokenizer.from_pretrained("Helsinki-NLP/opus-mt-en-de")
mbart50_tokenizer = MBart50Tokenizer.from_pretrained("facebook/mbart-large-50", src_lang="en_XX", tgt_lang="ro_RO")
pegasus_tokenizer = PegasusTokenizer.from_pretrained("google/pegasus-large")
reformer_tokenizer = ReformerTokenizer.from_pretrained("google/reformer-crime-and-punishment")
speech2text_tokenizer = Speech2TextTokenizer.from_pretrained("facebook/s2t-small-librispeech-asr")
t5_tokenizer = T5Tokenizer.from_pretrained("t5-small")
xlm_prophet_net_tokenizer = XLMProphetNetTokenizer.from_pretrained("microsoft/xprophetnet-large-wiki100-cased")
xlnet_tokenizer = XLNetTokenizer.from_pretrained("xlnet-base-cased")
```

I encoded and decoded `"hello world"` and got the following table:

| Tokenizer                    | Output                   |
| --- | --- |
| AlbertTokenizer              | 'hello world'            |
| BarthezTokenizer             | 'hello world'            |
| BertGenerationTokenizer      | 'hello world'            |
| BigBirdTokenizer             | '[CLS] hello world[SEP]' |
| CamembertTokenizer           | 'hello world'            |
| DebertaV2Tokenizer           | 'hello world'            |
| M2M100Tokenizer              | '__en__ hello world'     |
| MarianTokenizer              | '▁hello▁world'           |
| MBart50Tokenizer             | 'en_XX hello world'      |
| PegasusTokenizer             | 'hello world'            |
| ReformerTokenizer            | 'hello world'            |
| Speech2TextTokenizer         | 'hello world'            |
| T5Tokenizer                  | 'hello world\</s\>'        |
| XLMProphetNetTokenizer       | 'hello world[SEP]'       |
| XLNetTokenizer               | 'hello world\<sep\>\<cls\>'  |

I can hack my way through this using an `if tokenizer in problematic_sp_tokenizers: ...`, but it'll hurt my soul a bit 😉 

How do you think I should proceed from here?

Thanks for the opportunity to help,
Ben

Hi @NielsRogge 🙂 

I hope it's okay I'm tagging you, I just don't know if this issue is still relevant. If it is, I'll gladly work on it, and I'd love your opinion on the above message (the code was ran using Transformers v4.17.0.dev0).

Thanks,
Ben

Hi Ben!

Sorry forgot to reply. Great summary table over there! And as you can see, there's some work to do. 😂 

Feel free to open a PR!

Btw, `XLMRobertaTokenizer`'s corresponding checkpoint is xlm-roberta-base.

Great, will do!

Hi everyone!

I'm looking for some **Good first issues** for this project. It seems that the PR #15775 fixed this issue which can be **closed**.

I tested the snippet of code provided by @NielsRogge and it works fine!

Here the output:

```
2 [CLS]
10975 hello
126 world
3 [SEP]
```

* Python 3.9
* transformers 4.26.1
* sentencepiece 0.1.97

Thanks,
Damien G.

Thanks for flagging, closing this issue then :-)