RoBERTa ``tokenizer.decode`` does not produce the same sentence.
Author: flyaway1217
Pull request: no
Labels: name: wontfix

## Environment info
- `transformers` version: 3.0.2
- Platform: Linux-4.15.0-74-generic-x86_64-with-glibc2.27
- Python version: 3.8.0
- PyTorch version (GPU?): 1.5.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No.
- Using distributed or parallel set-up in script?: No.
     

### Who can help
@mfuntowicz
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao 
 examples/seq2seq: @sshleifer
 tensorflow: @jplu 
documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): RoBERTa

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

This code example should reproduce the issue:

```python3
from transformers import RobertaTokenizer
tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
s = """Meanwhile, Tucci's 'straight guy', the emphatic doctor Seger, is not developed into a more interesting character, like the fallible 'straight guys' Cuddy and Wilson."""
outputs = tokenizer(s)
input_ids = outputs['input_ids']
ss = tokenizer.decode(input_ids, skip_special_tokens=True)
print('s='+s)
print('ss='+ss)
```


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
I expect ``s`` and ``ss`` should be exactly the same. However, they are not. The outputs are:

```bash
s=Meanwhile, Tucci's 'straight guy', the emphatic doctor Seger, is not developed into a more interesting character, like the fallible 'straight guys' Cuddy and Wilson.

ss=Meanwhile, Tucci's'straight guy', the emphatic doctor Seger, is not developed into a more interesting character, like the fallible'straight guys' Cuddy and Wilson.
```
There are two spaces missing before ``'straight guy'``. 
I am not sure if this behavior is expected or it is a bug.
The thing is I want to use the sentence produced by the ``decode`` function and  I find the output is not exactly the same as the original sentence.

Thanks for the help!
<!-- A clear and concise description of what you would expect to happen. -->


Comments:
Seems to be an edge case from cleaning up tokenization on decoding :

https://github.com/huggingface/transformers/blob/c0b93a1c7a961e30b30d02d641c9d22120ef5d73/src/transformers/tokenization_utils_base.py#L2688

---

For this specific case, a work-around can be :

`ss = tokenizer.decode(input_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)`

---

But I think it's a bug. Is there any way to improve `clean_up_tokenization()` function ? 


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.