clean_up_tokenization_spaces=True won't clean up spaces
Author: MorenoLaQuatra
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.17.0
- Platform: Linux-5.10.0-051000-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.10.2+cu113 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes (the same on CPU)
- Using distributed or parallel set-up in script?: No


### Who can help
@LysandreJik, @Narsil, @SaulLu 


## Information

Model I am using (Bert, XLNet ...): BERT (`bert-large-cased`)

The problem arises when using:
* [x] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```python
encoded = tokenizer("This thing costs £4.56")
decoded = tokenizer.decode(encoded["input_ids"], clean_up_tokenization_spaces=True)
print (decoded)
```
Real output: `[CLS] This thing costs £4. 56 [SEP]`

I tried it also with NER pipelines and other text inputs.
Additional example: got `[CLS] ( including once - a - week tapping ) [SEP]` instead of `[CLS] (including once-a-week tapping) [SEP]`


## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Expected output: `[CLS] This thing costs £4.56 [SEP]`.

I expected the tokenizer to cleanup all the spaces introduced. Is there any different way to do so? Am I missing some trivial parameter?

Comments:
Hi @MorenoLaQuatra,

This is normal, as punctuation is considered as token splitter by this tokenizer. The `clean_up` is only meant to remove duplicate spaces, and normalize other space characters into the regular ascii one.

What are you trying to achieve ?

Using `.offset_mapping` is in general, could help you understand more what's going on without modifying the original string.

```python
from transformers import AutoTokenizer

sentence = "This thing costs £4.56"
tokenizer = AutoTokenizer.from_pretrained("bert-large-uncased")
encoded = tokenizer(sentence, return_offsets_mapping=True)
for _id, (start, stop) in zip(encoded.input_ids, encoded.offset_mapping):
    print(f"Token {_id} ({start}-{stop}) comes from {repr(sentence[start:stop])}")
```
```
Token 101 (0-0) comes from ''
Token 2023 (0-4) comes from 'This'
Token 2518 (5-10) comes from 'thing'
Token 5366 (11-16) comes from 'costs'
Token 1069 (17-18) comes from '£'
Token 2549 (18-19) comes from '4'
Token 1012 (19-20) comes from '.'
Token 5179 (20-22) comes from '56'
Token 102 (0-0) comes from ''
```

This allows you to understand what is happening without having to "recreate" the original string (`decode` cannot in general recover the original string)

Thank you for the explanation. I'm trying to build a custom NER-like system using token classification. I leverage offset_mapping during training to identify the token class. While trying to integrate it in a NER-like pipeline (TokenClassification) I found out it was generating entities from the tokenized version.

However, from your answer, I think it would be better to use char indexes to identify the correct span in the input sentence right?  

> However, from your answer, I think it would be better to use char indexes to identify the correct span in the input sentence right?

I cannot answer in general as I don't know if you have access to the original string for instance.
But if you do have access to it, then `offset_mapping` will be closer to what you expect most of the time. `decode` is a best effort way to represent what the model has seen, but it cannot in general output exactly what you sent in.

The reason is that `tokenizer.encode` is destructive and looses information. A simple example is that some tokenizer start by `.lower()` so we cannot in general recovering the capitalization. The same goes for spaces, `decode` will try to add them where they belong, but it cannot work 100% of the time.
Here around punctuation for instance You would like to get `"£4.56"` but you would want `"Hi. How are you doing ?"` (with the extra space after the dot). Since it looks the same to the input ids, `decode` has to make a choice and just use one form.

Ok, I got exactly your point. Thank you and sorry for the issue, I'll close it.

No worries, you're not alone having this issue, we're looking into how to communicate overall better the information.

@Narsil   I also have trouble in the same problem. And your explanation is informative. But is there any way to customize this tokenizer behaviors?