[`tokenizers`] Ensure that add_prefix_space is propagated to backend_tokenizer.pre_tokenizer
Author: tomaarsen
Pull request: yes

Hello!

## Pull Request overview
* Ensure that `add_prefix_space` is propagated to `backend_tokenizer.pre_tokenizer`

## What does this PR do?
This is an extrapolation of #35537, which was a fix to ensure that if `AutoTokenizer.from_pretrained("...", add_prefix_space=True)`, then the underlying [`ByteLevel` pre-tokenizer](https://huggingface.co/docs/tokenizers/en/api/pre-tokenizers#tokenizers.pre_tokenizers.ByteLevel) (at `backend_tokenizer.pre_tokenizer`) is also updated with `add_prefix_space=True`.

The PR is as simple as moving the patch, which looks like the following snippet, to the end of the `PreTrainedTokenizerFast.__init__` method.
```python
        pre_tok_state = json.loads(self.backend_tokenizer.pre_tokenizer.__getstate__())
        if pre_tok_state.get("add_prefix_space", add_prefix_space) != add_prefix_space:
            pre_tok_class = getattr(pre_tokenizers, pre_tok_state.pop("type"))
            pre_tok_state["add_prefix_space"] = add_prefix_space
            self.backend_tokenizer.pre_tokenizer = pre_tok_class(**pre_tok_state)

        self.add_prefix_space = add_prefix_space
```

The if-statement should only be True if the pre-tokenizer accepts "add_prefix_space" as an argument, and I assume that it's safer to override the pre_tokenizer with a new one than updating the `add_prefix_space` attribute on the pre-tokenizer, as it's all Rust-based behind the scenes.

I also added a test that should trigger for all tokenizers where the Rust-based tokenizer should be tested.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

Discussed with @ArthurZucker outside of GitHub.

## Who can review?

@ArthurZucker

- Tom Aarsen

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35593). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thank you @tomaarsen and I agree that this PR well works when the `pre_tokenizer` is simply `ByteLevel`. But I suspect that it does not work when the `pre_tokenizer` is a [`Sequence`](https://huggingface.co/docs/tokenizers/en/api/pre-tokenizers#tokenizers.pre_tokenizers.Sequence) including `ByteLevel`...

This was already not working before! 
But wait 1 week, we are merging stuff in `tokenizers` to just iterate of the Sequence! FYI @McPatate 😉