how to replace the existing token in a tokenizer
Author: muziyongshixin
Pull request: no

### Feature request

I have a tokenizer which have lots of preserved tokens like bellow:
```
 '<reserved_7>': 100,
 '<reserved_8>': 101,
 '<reserved_9>': 102,
 '<reserved_10>': 103,
 '<reserved_11>': 104,
 '<reserved_12>': 105,
 '<reserved_13>': 106,
 '<reserved_14>': 107,
```
I want to replace the '<reserved_7>' with '
' and  replace   '<reserved_8>' with '
'

what I want to get is a tokenizer which can act as below:
tokenizer.encode('
')   => 100


### Motivation

I want to replace the '<reserved_7>' with '
' and  replace   '<reserved_8>' with '
'


### Your contribution

no

Comments:
Hi @muziyongshixin, thanks for raising an issue!

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

To learn about how to how to modify the tokenizers, you can check out the documentation, [1](https://huggingface.co/docs/transformers/v4.36.0/en/main_classes/tokenizer#tokenizer), [2](https://huggingface.co/docs/transformers/v4.36.0/en/main_classes/tokenizer#transformers.PreTrainedTokenizer.add_tokens). For example, you can add tokens to the tokenzers vocabulary by using the [`add_tokens` method](https://huggingface.co/docs/transformers/v4.36.0/en/main_classes/tokenizer#transformers.PreTrainedTokenizer.add_tokens). 

> Hi @muziyongshixin, thanks for raising an issue!
> 
> This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.
> 
> To learn about how to how to modify the tokenizers, you can check out the documentation, [1](https://huggingface.co/docs/transformers/v4.36.0/en/main_classes/tokenizer#tokenizer), [2](https://huggingface.co/docs/transformers/v4.36.0/en/main_classes/tokenizer#transformers.PreTrainedTokenizer.add_tokens). For example, you can add tokens to the tokenzers vocabulary by using the [`add_tokens` method](https://huggingface.co/docs/transformers/v4.36.0/en/main_classes/tokenizer#transformers.PreTrainedTokenizer.add_tokens).

thanks for your  reply@amyeroberts 

I know the `add_tokens` function, but I don't want to change the vocabulary size, this will introduce some inconvience in finetuning process(like resize the embedding layer and the lm_head).
So I am wondering whether there is a method to replace the existing tokens.

P.S. I am using a sentencepiece tokenizer which is released in [this baichuan2 repo](https://github.com/baichuan-inc/Baichuan2) 


@ArthurZucker will be best placed to answer this. I believe `add_tokens` or `add_special_tokens` is the safe way to do this, as it correctly keeps track of the modifications vs. from the loaded checkpoint and ensures the correct preprocessing is applied before mapping to the token id. It should be possible to define your own vocab files to load a tokenizer but I don't know about the guarantees of the behaviour vs the original tokenizer. 

hey! You should modify manually both the `added_tokens_decoder` field (saved in the `tokenizer_config.json` ) and the `added_tokens` field (saved in the `tokenizer.json`). We don't really support this manually, but that is the recommended way to do it! (If the reserved tokens were already part of the vocab, so not AddedTokens, then you have to overwrite the vocab as well, the vocab files, to make sure they are removed) that would be hard than if it's just the content of the added tokens that you are trying to modify 😉 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I know this is an old discussion but i think there should be a way to edit tokenizers vocab programmatically. It just feels intuitive for such a functionality to be present