Mask Token Spacing in RobertaTokenizer
Author: mpoemsl
Pull request: no
Labels: name: bug

### System Info


- `transformers` version: 4.18.0
- Platform: Linux-5.16.13-200.fc35.x86_64-x86_64-with-glibc2.34
- Python version: 3.10.2
- Huggingface_hub version: 0.5.1
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no



### Who can help?

@SaulLu 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from transformers import RobertaTokenizer, BertTokenizer

bert_tk = BertTokenizer.from_pretrained("bert-base-cased")
roberta_tk = RobertaTokenizer.from_pretrained("roberta-base")

enc_bert = bert_tk("Testing the spacing of " + bert_tk.pad_token + " and " + bert_tk.mask_token + " tokens.")
enc_roberta = roberta_tk("Testing the spacing of " + roberta_tk.pad_token + " and " + roberta_tk.mask_token + " tokens.")

dec_bert = bert_tk.decode(enc_bert.input_ids, skip_special_tokens=False)
dec_roberta = roberta_tk.decode(enc_roberta.input_ids, skip_special_tokens=False)

print("bert", dec_bert)
print("roberta", dec_roberta)
```

Output is:

```
bert [CLS] Testing the spacing of [PAD] and [MASK] tokens. [SEP]
roberta 
Testing the spacing of <pad> and<mask> tokens.

```

### Expected behavior


I expected Roberta's `<mask>` token to be surrounded by spaces as it is done with the `<pad>` token. In BERT, this is done the same way for both special tokens - why not in Roberta?

If someone confirms that this is indeed a bug and not expected behavior, I would be happy to try to get to the root cause myself.



Comments:
Hello @mpoemsl ,

Sorry for the late reply. I don't think it's really a bug but I'd be interested to know why you used the decoding feature with Roberta (and Bert). :smile: 

Hi @SaulLu, thanks for your reply! 

I have been working on a model that uses representations derived from Transformer language models (e.g. RoBERTa, BERT) and that is trained on annotations on word, token, and character level, so I had to do a lot of mapping between those granularities and was inspecting them with `decode`.

The `RobertaTokenizer` has shown a lot of behavior that to me was unexpected, so I am mostly sticking with BERT for now. Another example is the `add_prefix_space=True` behavior:

```python
from transformers import RobertaTokenizer

tk = RobertaTokenizer.from_pretrained("roberta-base", add_prefix_space=True)
enc = tk(["Testing", "spacing", "lorem", "ipsum", "abc"],  return_tensors="pt", is_split_into_words=True)
dec = tk.batch_decode(enc.input_ids.T)

print(dec)
```

Output is:

```
['
', ' Testing', ' spacing', ' lore', 'm', ' ', 'ips', 'um', ' ab', 'c', '
']
```

What is surprising to me here is that it inserted a literal space token before `'ips'` and not a space-prefixed `' ips'`. It probably has to with whether the space-prefixed token is in the vocab or not.

I agree that those are just quirks of RoBERTa and not really bugs, but should this behavior perhaps be noted in the documentation somewhere? The [current docs on RobertaTokenizer](https://huggingface.co/docs/transformers/model_doc/roberta#transformers.RobertaTokenizer) do not mention the distinction of whether the space-prefixed version of a token is in the vocab and they do not mention the spacing behavior of special tokens.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.