Inconsistencies between `.save_pretrained` and `from_pretrained` for slow and fast tokenizers (RoFormer)
Author: xenova
Pull request: no

### System Info

- `transformers` version: 4.35.2
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.0+cu121 (False)
- Tensorflow version (GPU?): 2.15.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.7.5 (cpu)
- Jax version: 0.4.20
- JaxLib version: 0.4.20
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

My original problem occurred when loading and saving with AutoTokenizer:
```py
from transformers import AutoTokenizer
# Load original tokenizer
original = AutoTokenizer.from_pretrained('alchemab/antiberta2')
print(original("生活的真谛是"))
# {'input_ids': [1, 4, 4, 4, 4, 4, 4, 2], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1]}

# Save tokenizer
original.save_pretrained('saved')

# Load this new tokenizer
new = AutoTokenizer.from_pretrained('saved')
print(new("生活的真谛是"))
# {'input_ids': [1, 4, 2], 'token_type_ids': [0, 0, 0], 'attention_mask': [1, 1, 1]}
```

Digging a bit deeper, it seems to be an issue with the slow to fast converter, with certain default values being overridden (presumably `handle_chinese_chars` in `BertNormalizer`). I know RoFormer isn't a very popular model these days, but since it uses a near-identical tokenization strategy to Bert models, this issue may have implications elsewhere.

### Expected behavior

Should produce the same (correct) results if it were loaded with the original (slow) tokenizer

```py
from transformers import RoFormerTokenizer
# Load original tokenizer
original = RoFormerTokenizer.from_pretrained('alchemab/antiberta2')
print(original("生活的真谛是"))
# {'input_ids': [1, 4, 4, 4, 4, 4, 4, 2], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1]}

# Save tokenizer
original.save_pretrained('saved')

# Load this new tokenizer
new = RoFormerTokenizer.from_pretrained('saved')
print(new("生活的真谛是"))
# {'input_ids': [1, 4, 4, 4, 4, 4, 4, 2], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1]}
```

Comments:
Another weird thing is that behaviour changes after saving the tokenizer:
```py
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained('alchemab/antiberta2')
>>> text='test $1 R2 #3 €4 £5 ¥6 ₣7 ₹8 ₱9 test'
>>> tokenizer.encode(text)
[1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 2]
>>> len(tokenizer.encode(text))
21
>>> tokenizer.save_pretrained('saved')
('saved/tokenizer_config.json', 'saved/special_tokens_map.json', 'saved/vocab.txt', 'saved/added_tokens.json', 'saved/tokenizer.json')
>>> tokenizer.encode(text)
[1, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 2]
```


Alright, there is a discrepency in the `tokenization_roformer_fast`, the `__setstate__` function and `__getstate__` function will both set a different tokenizer. Save-pretrained will set another one.
That is not consistent but I have no Idea why this is done this way. 

https://github.com/huggingface/transformers/blob/0cdcd7a2b319689d75ae4807cfb7b228aa322f83/src/transformers/models/roformer/tokenization_roformer_fast.py#L137-L145

and: 

https://github.com/huggingface/transformers/blob/0cdcd7a2b319689d75ae4807cfb7b228aa322f83/src/transformers/models/roformer/tokenization_roformer_fast.py#L204-L214 

https://github.com/huggingface/tokenizers/issues/581 explains why 😉 But it was not well done in this specific case IMO

1. First case we have a conversion from slow
2. We don't and we use `fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)` after that we never set the correct custom pre_tokenizer

Using `from_slow = True` pretty much fixes the issue, this should also fix it
```python 
        # Make sure we correctly set the custom PreTokenizer
        vocab = self.backend_tokenizer.get_vocab()
        self.backend_tokenizer.pre_tokenizer = PreTokenizer.custom(JiebaPreTokenizer(vocab))
```




Thanks so much @ArthurZucker for the investigation and fix!