DeBERTa-v3 does not preserve spaces before/after additional special tokens in convert_tokens_to_string output
Author: JohnGiorgi
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.12.5
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.8.5
- PyTorch version (GPU?): 1.10.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No.
- Using distributed or parallel set-up in script?: No.

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- ALBERT, BERT, XLM, DeBERTa, DeBERTa-v2, ELECTRA, MobileBert, SqueezeBert: @LysandreJik
- encoder-decoder models (For example, BlenderBot, BART, Marian, Pegasus, T5, ByT5): @patrickvonplaten, @patil-suraj
- Longformer, Reformer, TransfoXL, XLNet, FNet: @patrickvonplaten
- FSMT: @stas00
- Funnel: @sgugger
- GPT-2, GPT: @patrickvonplaten, @LysandreJik
- RAG, DPR: @patrickvonplaten, @lhoestq
- TensorFlow: @Rocketknight1
- JAX/Flax: @patil-suraj @patrickvonplaten 
- TAPAS, LayoutLM, LayoutLMv2, LUKE, ViT, BEiT, DEiT, DETR, CANINE: @NielsRogge
- GPT-Neo, GPT-J, CLIP: @patil-suraj
- Wav2Vec2, HuBERT, SpeechEncoderDecoder: @patrickvonplaten, @anton-l

If the model isn't in the list, ping @LysandreJik who will redirect you to the correct contributor.

Library:

- Benchmarks: @patrickvonplaten
- Deepspeed: @stas00
- Ray/raytune: @richardliaw, @amogkam
- Text generation: @patrickvonplaten
- Tokenizers: @LysandreJik
- Trainer: @sgugger
- Pipelines: @Narsil
- Speech: @patrickvonplaten, @anton-l
- Vision: @NielsRogge, @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model, report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

For research projetcs, please ping the contributor directly. For example, on the following projects:

- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

@LysandreJik

## Information

Model I am using (Bert, XLNet ...): `microsoft/deberta-v3-small`

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Initialize a DeBERTa-v3 tokenizer with `additional_special_tokens`.
2. Tokenize some text with `tokenize` that contains one or more of those special tokens.
3. Attempt to convert the tokens to a string with `convert_tokens_to_string`
4. DeBERTa-v3 does not include a space before/after the special token in the resulting string. BERT (and earlier versions of DeBERTa) do.

```python
from transformers import AutoTokenizer, AutoModel

special_tokens = ["<SPECIAL>"]
text = "some text with an additional special token <SPECIAL>"

# BERT
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased", additional_special_tokens=special_tokens)
print(tokenizer.convert_tokens_to_string(tokenizer.tokenize(text)))
# => some text with an additional special token <SPECIAL>

# DeBERTa (original)
tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-base", additional_special_tokens=special_tokens)
print(tokenizer.convert_tokens_to_string(tokenizer.tokenize(text)))
# => some text with an additional special token <SPECIAL>

# DeBERTa (v3)
tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-v3-small", additional_special_tokens=special_tokens)
print(tokenizer.convert_tokens_to_string(tokenizer.tokenize(text)))
# => some text with an additional special token<SPECIAL>
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

I expect that spaces before/after any special tokens added with `additional_special_tokens` will be preserved when calling `tokenizer.convert_tokens_to_string(tokenizer.tokenize(text))`.


Comments:
Sorry for the delay in answering this, pinging @SaulLu so she can take a look when she has the time :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@LysandreJik @SaulLu This still happens on the latest version of Transformers and with the latest version of DeBERTa-v3, so I am commenting to keep it open.

Thank you very much for the detailed issue! 

Indeed, you have put your finger on an inconsistency: what is happening is that the slow and fast tokenizer of DeBERTa (original) do not behave in the same way:
```python
tokenizer = AutoTokenizer.from_pretrained(
    "microsoft/deberta-base", 
    additional_special_tokens=special_tokens, 
    use_fast=False
)
print(f"Output with {type(tokenizer)}:\n", tokenizer.convert_tokens_to_string(tokenizer.tokenize(text)))
# => Output with <class 'transformers.models.deberta.tokenization_deberta.DebertaTokenizer'>:  
# some text with an additional special token<SPECIAL>

# DeBERTa (original) fast
tokenizer = AutoTokenizer.from_pretrained(
    "microsoft/deberta-base", 
    additional_special_tokens=special_tokens, 
    use_fast=True
)
print(f"Output with {type(tokenizer)}:\n", tokenizer.convert_tokens_to_string(tokenizer.tokenize(text)))
# => Output with <class 'transformers.models.deberta.tokenization_deberta_fast.DebertaTokenizerFast'>: 
# some text with an additional special token <SPECIAL>
```
As a result the issue seems more linked to the the workflow of the slow tokenizers.

However, finding the right way to fix the problem is less obvious because:
- `convert_tokens_to_string` is used in `_decode` of `PreTrainedTokenizer` (the base class of all slow tokenizers)
- `DebertaTokenizer` (original) inherits from `GPT2Tokenizer` where `convert_tokens_to_string` is defined
- `DebertaV2Tokenizer`  uses a different strategy than `GPT2Tokenizer` to implement `convert_tokens_to_string`

To get an broader view of the problem, could you share with us what your use case is for this command (what do you want to see with it? Is it manual work? in production?)?

Thanks for the detailed response @SaulLu!

I have a task where I need to add special tokens to the text to introduce some structure. A common use case of this are the "marker tokens" used in named relation extraction. A simplified example is:

```python
text = "<ORG> Apple </ORG> is looking at buying <GPE> U.K. </GPE> startup for <MONEY> $1 billion </MONEY>"
```

Ideally, we could add all these tokens as `additional_special_tokens` so they don't get split. Indeed, it works fine with BERT and the original DeBERTa, so I was curious as to why it doesn't work with DeBERTa V3.

Thank you very much for your answer! Very interesting use case! 

And in particular, why on this use case do you need to use `tokenizer.convert_tokens_to_string(tokenizer.tokenize(text)`?

For DeBERTa (original and V3), I guess the `tokenizer.decode(tokenizer.encode(text)` command should give the result you were expecting initially. :blush:  

Ahhh, `tokenizer.decode(tokenizer.encode(text)` does work! And it works for BERT as well. 

There was no specific reason to use `convert_tokens_to_string`, I just thought that would be the correct method to use! Thanks for the tip with `tokenizer.decode(tokenizer.encode(text)`

Actually, I now remember why I wanted to use `convert_tokens_to_string`. In the case of an autoregressive decoder, generating some output token by token, that may include some special tokens. I would like to recover from output a string, which maintains the expected spaces around special tokens. Here is a simplified example:

```python
special_tokens = ["<DISEASE>", "</DISEASE>", "<DRUG>", "</DRUG>"]
text = "<DISEASE> Anaphylaxis </DISEASE> to <DRUG> cisplatin </DRUG> is an infrequent life-threatening complication"

tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-v3-small", additional_special_tokens=special_tokens)

# Tokenize the text to mimic what a decoder would have generated, token-by-token
decoder_output = tokenizer.tokenize(text)
print(decoder_output)
# => ['<DISEASE>', '▁Ana', 'phyl', 'axis', '</DISEASE>', '▁to', '<DRUG>', '▁cisplatin', '</DRUG>', '▁is', '▁an', '▁infrequent', '▁life', '-', 'threatening', '▁complication']
# Try to go backwards
print(tokenizer.convert_tokens_to_string(decoder_output))
# => <DISEASE> Anaphylaxis</DISEASE> to<DRUG> cisplatin</DRUG> is an infrequent life-threatening complication
```

Which doesn't produce the correct spacing. I can solve that using the `decode(encode())` strategy

```python
print(tokenizer.decode(tokenizer.encode(tokenizer.convert_tokens_to_string(decoder_output), add_special_tokens=False)))
# => <DISEASE> Anaphylaxis </DISEASE> to <DRUG> cisplatin </DRUG> is an infrequent life-threatening complication
```

I guess the only downside is that you have to call 3 (!) `tokenizer` methods to get the job done (`decode`, `encode` and `convert_tokens_to_string`).

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.