inconsistent BertTokenizer and BertTokenizerFast
Author: idiomer
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.12.5
- Platform: centos
- Python version: 3.7.6
- PyTorch version (GPU?): CPU
- Tensorflow version (GPU?): CPU
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


## Problem
BertTokenizer and BertTokenizerFast behave differently for unknown tokens

``` python
bert_tokenizer = BertTokenizer.from_pretrained('bert-base-chinese')
bert_tokenizer_fast = BertTokenizerFast.from_pretrained('bert-base-chinese')

content_encodings = bert_tokenizer(text_list, truncation=True, padding='max_length', max_length=64, return_tensors='pt')
content_encodings_fast = bert_tokenizer_fast(text_list, truncation=True, padding='max_length', max_length=64, return_tensors='pt')
```


![image](https://user-images.githubusercontent.com/17400651/146740197-9b549225-f717-428c-89ed-76cbdd297e3a.png)



Comments:
- `transformers` version: 4.14.1
- Platform: Darwin 20.3.0
- Python version: 3.8.6
- PyTorch version (GPU?): 1.10.1, no GPU

Using the below code, I was not able to reproduce the differences you show. The two tokenizers resulted in the same output. Maybe upgrade `transformers` and clear your `transformers` cache (default is `~/.cache/huggingface/transformers`)?

Code:

<details>

```
import torch
from transformers import BertTokenizer, BertTokenizerFast


bert_tokenizer = BertTokenizer.from_pretrained('bert-base-chinese')
bert_tokenizer_fast = BertTokenizerFast.from_pretrained('bert-base-chinese')
print(f'bert_tokenizer == bert_tokenizer_fast: {bert_tokenizer == bert_tokenizer_fast}\n')

strings = ['早安 都没人聊天吗🥱', '给大家表演下靓仔打哈欠🥱', '📷每周总有那么两天是开心的〜好起来了🤏']

for string in strings:
    print('-'*100, '\n')
    print(string)
    content_encodings = bert_tokenizer(string, truncation=True, padding='max_length', max_length=64, return_tensors='pt')
    content_encodings_fast = bert_tokenizer_fast(string, truncation=True, padding='max_length', max_length=64, return_tensors='pt')

    decoded_slow = bert_tokenizer.decode(content_encodings['input_ids'][0].tolist())
    decoded_fast = bert_tokenizer_fast.decode(content_encodings_fast['input_ids'][0].tolist())

    print('Slow:\n', decoded_slow, sep='')
    print('Fast:\n', decoded_fast, sep='')

    print(f'\ndecoded_slow == decoded_fast: {decoded_slow == decoded_fast}\n')
```

</details>

Output:

<details>

```
bert_tokenizer == bert_tokenizer_fast: False

---------------------------------------------------------------------------------------------------- 

早安 都没人聊天吗🥱

Slow:
[CLS] 早 安 都 没 人 聊 天 吗 [UNK] [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]
Fast:
[CLS] 早 安 都 没 人 聊 天 吗 [UNK] [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]

decoded_slow == decoded_fast: True

---------------------------------------------------------------------------------------------------- 

给大家表演下靓仔打哈欠🥱

Slow:
[CLS] 给 大 家 表 演 下 靓 仔 打 哈 欠 [UNK] [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]
Fast:
[CLS] 给 大 家 表 演 下 靓 仔 打 哈 欠 [UNK] [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]

decoded_slow == decoded_fast: True

---------------------------------------------------------------------------------------------------- 

📷每周总有那么两天是开心的〜好起来了🤏

Slow:
[CLS] [UNK] 每 周 总 有 那 么 两 天 是 开 心 的 〜 好 起 来 了 [UNK] [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]
Fast:
[CLS] [UNK] 每 周 总 有 那 么 两 天 是 开 心 的 〜 好 起 来 了 [UNK] [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]

decoded_slow == decoded_fast: True
```

</details>

> * `transformers` version: 4.14.1
> * Platform: Darwin 20.3.0
> * Python version: 3.8.6
> * PyTorch version (GPU?): 1.10.1, no GPU
> 
> Using the below code, I was not able to reproduce the differences you show. The two tokenizers resulted in the same output. Maybe upgrade `transformers` and clear your `transformers` cache (default is `~/.cache/huggingface/transformers`)?
> 
> Code:
> 
> Output:


I ran the above code, the problem still exist.

Therefore, I changed to use a new gpu mechine that was newly installed with `torch.__version__ = 1.10.1+cu102, transformers.__version__ = 4.14.1, python = 3.7.3`, and manually downloaded bert-base-chinese model from transformers [repo](https://huggingface.co/bert-base-chinese) to local.  Until all things were ready, I ran the above code again, the slow and the fast still show difference. Here is the screenshot

![image](https://user-images.githubusercontent.com/17400651/146858325-db73cea2-422b-40be-8657-3195b665b02d.png)

Here is the Output:
```
torch.__version__ = 1.10.1+cu102, transformers.__version__ = 4.14.1
bert_tokenizer == bert_tokenizer_fast: False

---------------------------------------------------------------------------------------------------- 

早安 都没人聊天吗🥱
Slow:
[CLS] 早 安 都 没 人 聊 天 吗 [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]
Fast:
[CLS] 早 安 都 没 人 聊 天 吗 [UNK] [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]

decoded_slow == decoded_fast: False

---------------------------------------------------------------------------------------------------- 

给大家表演下靓仔打哈欠🥱
Slow:
[CLS] 给 大 家 表 演 下 靓 仔 打 哈 欠 [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]
Fast:
[CLS] 给 大 家 表 演 下 靓 仔 打 哈 欠 [UNK] [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]

decoded_slow == decoded_fast: False

---------------------------------------------------------------------------------------------------- 

📷每周总有那么两天是开心的〜好起来了🤏
Slow:
[CLS] [UNK] 每 周 总 有 那 么 两 天 是 开 心 的 〜 好 起 来 了 [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]
Fast:
[CLS] [UNK] 每 周 总 有 那 么 两 天 是 开 心 的 〜 好 起 来 了 [UNK] [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]

decoded_slow == decoded_fast: False
```

Thanks for raising the issue! Let me ping @SaulLu, who might have better context on what's happening

I can confirm that I was able to reproduce this issue on a [fresh Colab notebook](https://colab.research.google.com/drive/1KqN09oSEyzyQc1brEmaNpApUzIH76qy-?usp=sharing). Also, the problem is not specific to `bert-base-chinese`, as `bert-base-uncased` also exhibits the same phenomenon. 

Code:

```python
import torch
from transformers import BertTokenizer, BertTokenizerFast

model_name = "bert-base-uncased"
bert_tokenizer = BertTokenizer.from_pretrained(model_name)
bert_tokenizer_fast = BertTokenizerFast.from_pretrained(model_name)
print(f'bert_tokenizer == bert_tokenizer_fast: {bert_tokenizer == bert_tokenizer_fast}\n')

strings = ['🥱', '📷🤏', '🦾']

for string in strings:
    print('-'*100, '\n')
    print(string)
    content_encodings = bert_tokenizer(string, truncation=True, padding='max_length', max_length=64, return_tensors='pt')
    content_encodings_fast = bert_tokenizer_fast(string, truncation=True, padding='max_length', max_length=64, return_tensors='pt')

    decoded_slow = bert_tokenizer.decode(content_encodings['input_ids'][0].tolist())
    decoded_fast = bert_tokenizer_fast.decode(content_encodings_fast['input_ids'][0].tolist())

    print('Slow:\n', decoded_slow, sep='')
    print('Fast:\n', decoded_fast, sep='')

    print(f'\ndecoded_slow == decoded_fast: {decoded_slow == decoded_fast}\n')
```

Output:

```
bert_tokenizer == bert_tokenizer_fast: False

---------------------------------------------------------------------------------------------------- 

🥱
Slow:
[CLS] [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]
Fast:
[CLS] [UNK] [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]

decoded_slow == decoded_fast: False

---------------------------------------------------------------------------------------------------- 

📷🤏
Slow:
[CLS] [UNK] [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]
Fast:
[CLS] [UNK] [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]

decoded_slow == decoded_fast: True

---------------------------------------------------------------------------------------------------- 

🦾
Slow:
[CLS] [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]
Fast:
[CLS] [UNK] [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]

decoded_slow == decoded_fast: False
```

Thank you very much for the clarity of the examples!

I'm currently on vacation, so I can only look deeper into this issue when I come back next week :slightly_smiling_face: 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.