Access to pre_tokenizer for PreTrainedTokenizer
Author: GitMew
Pull request: no

### Feature request

Give access to setting a `pre_tokenizer` for a `transformers.PreTrainedTokenizer`, similar to how this works for `PreTrainedTokenizerFast`.


### Motivation

As far as I understand from [these docs](https://huggingface.co/docs/transformers/v4.33.2/en/main_classes/tokenizer), there are two interfaces for interacting with tokenizers in the HuggingFace ecosystem: `PreTrainedTokenizerFast` is a wrapper around Rust code, and `PreTrainedTokenizer` is supposed to be the slow Python equivalent.

`PreTrainedTokenizerFast` has a property `backend_tokenizer` which is a `tokenizers.Tokenizer` object, which has a `pre_tokenizer` property and is built from a `tokenizers.models.Model` subclass (the thing that does the tokenization). You can instantiate a `PreTrainedTokenizerFast` from such a `Tokenizer` object with the constructor argument `tokenizer_object`. Meanwhile, none of this is accessible for a `PreTrainedTokenizer`. 

Here is my use-case: I have a function `tokenizeWord(w: str)` implemented entirely in Python to segment a single word into subwords. I would now like to 
1. Build a `PreTrainedTokenizer` from this function, and
2. pre-tokenize sentences on punctuation and whitespace so that each word is sent to that function separately.

I can do the first as follows (at least I think this is how it's supposed to be done):
```
class CustomTokenizer(PreTrainedTokenizer):

    def __init__(self, custom_tkz_algorithm, **kwargs):
        super().__init__(**kwargs)
        self.algorithm = custom_tkz_algorithm
        self.vocab         = self.algorithm.get_vocab()
        self.reverse_vocab = {i: s for s,i in self.vocab.items()}  # Assume that the vocabulary is injective (no duplicate IDs)

    @property
    def vocab_size(self) -> int:
        return len(self.vocab)

    def _convert_token_to_id(self, token):
        return self.vocab[token]

    def _convert_id_to_token(self, index: int) -> str:
        return self.reverse_vocab[index]

    def _tokenize(self, text, **kwargs) -> List[str]:
        """
        Converts a string in a sequence of tokens (string), using the tokenizer. Split in words for word-based
        vocabulary or sub-words for sub-word-based vocabularies (BPE/SentencePieces/WordPieces).

        Do NOT take care of added tokens.
        """
        return tokenizeWord(text)
```

but where does the pre-tokenizer come in? It doesn't even seem feasible to manually use the pre-tokenizers provided by `tokenizers.pre_tokenizers` (e.g. `Whitespace`, to name one) because those all provide Rust interfaces and hence the objects they output don't work with a simple string segmentation function.

### Your contribution

None.

Comments:
Hey! The equivalent of `pre-tokenizers` is not impelemted directly for `PretrainedTokenizers` (yet, something might be on it's way). The pre-tokenization is usually done in the `prepare_for_tokenization` function.

@ArthurZucker Thanks for your reply! That's unfortunate. One would expect that the two classes derive from the same base class and that that base class offers pretokenisation (and postprocessing, while we're at it).

I did see the `prepare_for_tokenization` function, but as far as I can see, it is supposed to output a string, not e.g. a list of strings to be tokenised separately, unless I violate its type signature. That seems like a bad idea, given that the `PreTrainedTokenizer.tokenize` function looks something like this, abstracted:

```
    def tokenizer(text, **kwargs):
        text, kwargs = self.prepare_for_tokenization(text, **kwargs)
        ...
        tokens = self.tokens_trie.split(text)
        ...
        tokenized_text = []
        for token in tokens:
            ...
            tokenized_text.extend(self._tokenize(token))
```
...wherein I assume the `tokens_trie` is only used to isolate a small set of very special tokens, and `.split` expects a string. Do you have an example of how people would e.g. include `Whitespace()` in `prepare_for_tokenization` compatible with this?

Usually the `Whitespace()` is done in this function, which is applied to all the inputs if the input is a batch of strings. 
A lot of sentencepiece models do this (see LlamaTokenizer)  for example. It is sometimes done in the `tokenize`. 
 I agree with you that the fast and slow lack consistency, and note this for futur improvements 🤗 Thanks for your input

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.