basic_tokenizer don't preserve token encoding/format
Author: valtheval
Pull request: no

Hello all!

## Environment info
- `transformers` version: 4.3.2
- Platform: Linux-4.19.0-13-cloud-amd64-x86_64-with-glibc2.28
- Python version: 3.8.6
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No
- the code was run on jupyter notebook


### Who can help
@LysandreJik


## Issue
I have the following code
```
model = 'microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract-fulltext'
tokenizer = BertTokenizer.from_pretrained(model, do_lower_case=False)

s = 'View and Print-FDA Drug Safety Communication: FDA requiring color changes to Duragesic (fentanyl) pain patches to aid safetyâ\x80\x95emphasizing that accidental exposure to used patches can cause death'
```
When using `basic_tokenizer` it changes the token (not the same forme (encoding) of the original sentence)

```
tokenizer.basic_tokenizer.tokenize(s)
>>> ['View', 'and', 'Print', '-', 'FDA', 'Drug', 'Safety', 'Communication', ':', 'FDA', 'requiring', 'color', 'changes', 'to', 'Duragesic', '(', 'fentanyl', ')', 'pain', 'patches', 'to', 'aid', 'safetyâemphasizing', 'that', 'accidental', 'exposure', 'to', 'used', 'patches', 'can', 'cause', 'death']
```
the original token `safetyâ\x80\x95emphasizing` is tokenized in  `safetyâemphasizing`

**2 issues then:**
- Is this the normal behavior ? it seems not or I am using it wrongly
- It seems there is no documentation about basic_tokenizer object in the huggingface documentation

Any help/explanation would be welcomed :)



Comments:
Hello! The basic tokenizer is only a building block of the `BertTokenizer` - and it was not intended to be used independently.

What are you trying to achieve especially, that the `BertTokenizer` cannot?

Usually, it is best to assume that the tokenizer should be used as it is configured on the hub - as it is with that tokenizer that the model was trained, and staying consistent is important to obtain good results.

Hi thanks for the reply.

My original problem is that I want to decode bert token ids (int --> word)
If I use BertTokenizer it sometimes generates [UNK] token which can't be decode back to the original token (the real word that produced the [UNK] token). I then use basic_tokenizer to have the list of raw token and replace [UNK] by the right token using its index in the sentence. But I am facing inconsistancies

Here is an example:
```
raw_sent = "patches to aid safetyâ\x80\x95emphasizing"
ids = bert_tokenizer.encode(raw_sent)
tokens = bert_tokenizer.decode(ids)
print(ids)
print(tokens)
```
gives:
```
[2, 13453, 1701, 6974, 1, 3]
'[CLS] patches to aid [UNK] [SEP]'
```
In my pipeline I receive the raw sentence and the list of ids (int), I want to figure out wich word in the sentence produced the [UNK] token.

I do:
```
basic_token = bert_tokenizer.basic_tokenizer.tokenize(raw_sent)
print(basic_token)
['patches', 'to', 'aid', 'safetyâemphasizing']
```
So I know that id `1` in the list `[2, 13453, 1701, 6974, 1, 3]` corresponds to `safetyâemphasizing`. So here is the problem: `safetyâemphasizing` is different from `safetyâ\x80\x95emphasizing` in the original sentence which leads to further errors in the following of the pipeline (especially finding de spans of the token)


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.