Fix added-token prefix space in SentencePiece fast tokenizers
Author: qflen
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=45822)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=45822)
<!-- ci-dashboard-badge:end -->

Fixes #28218

When a user adds a token to a SentencePiece-style fast tokenizer (e.g. NLLB), the tokens-trie splits the input
at the new token and runs the Metaspace pre_tokenizer on each chunk. 

The default `prepend_scheme="always"` then prefixes every chunk with `▁`, so `"abcdgym"` tokenizes to `["abcd", "▁gym"]` and `decode(encode("abcdgym"))` returns `"abcd gym"`.

This PR mirrors what `SpmConverter.pre_tokenizer` already does for non-legacy spm tokenizers: 
when a post-init, non-special `add_tokens` call lands on a standalone `Metaspace(prepend_scheme="always")`, the pre_tokenizer is swapped for one with `prepend_scheme="first"` so only the first chunk receives the prefix.

Sequence pipelines (WhitespaceSplit + Metaspace, used by T5, Pegasus, UDOP, LASR) are left untouched because they rely on `"always"` for whitespace handling.

With `tokenizer.add_tokens(["abcd"])` applied to a fresh NLLB tokenizer:

| Input | Before | After |
| --- | --- | --- |
| `"abcdgym"` | `["abcd", "▁gym"]` →  `"abcd gym"` | `["abcd", "gym"]` → `"abcdgym"` |
| `"I like to walk abcdgym along the beach"` | `[..., "▁walk", "▁", "abcd", "▁gym", "▁along", ...]` → `"I like to walk abcd gym along the beach"` | `[..., "▁walk", "▁", "abcd", "gym", "▁along", ...]` → `"I like to walk abcdgym along the beach"` |
| `"abcd gym"` (literal space already there) | `["abcd", "▁gym"]` | `["abcd", "▁gym"]` (unchanged) |

Inputs without any added tokens are unchanged (e.g. `"Hello world"` → `["▁Hello", "▁world"]` either way).

## Code Agent Policy

- [x] I confirm that this is not a pure code agent PR.

## Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the contributor guideline, Pull Request section?
- [x] Was this discussed or approved via a Github issue or the forum? Please add a link to it if that is the case.
  Issue: https://github.com/huggingface/transformers/issues/28218
  (`@ArthurZucker` proposed mirroring the SpmConverter approach)
- [ ] Did you make sure to update the documentation with your changes?
- [x] Did you write any new necessary tests?

## Who can review?

`@ArthurZucker` `@itazap`

---

## AI assistance disclosure

I've used Opus 4.7 Max Effort for debugging and reviewed / double-checked every line.

- Coordination: 
https://github.com/huggingface/transformers/issues/28218  (maintainer guidance to follow `SpmConverter.pre_tokenizer`
  
- Claimed issue in:
  https://github.com/huggingface/transformers/issues/28218#issuecomment-2050562552)

- Why this is not a duplicate:  
The only related open PR (#31315) targets a different issue (#30824, plumbing `add_prefix_space` through the fast tokenizer); it does not touch the post-init alignment of `prepend_scheme` for added tokens.

- Tests run:
  - `make style` and `make typing` and `make fix-repo`, all green, no extra diff
  - `pytest tests/tokenization/test_tokenization_fast.py` (11 passed, 3 skipped)
  - `pytest tests/test_tokenization_common.py` (18 passed, 3 skipped)
  - `pytest tests/models/nllb/test_tokenization_nllb.py` (61 passed, 4 skipped)
  - `pytest tests/models/big_bird/test_tokenization_big_bird.py` (100 passed, 4 skipped)
  - `pytest tests/models/{reformer,t5,albert,udop,moshi,seamless_m4t}/test_tokenization_*.py` (all green)
  - The new regression test `test_added_token_does_not_get_metaspace_prefix` was verified to fail on main
    (without the fix) and pass with the fix.

Comments:
cc @arthurzucker @itazap

okay but since "always" and "first" are equivalent for normal text (without added tokens), the root fix is just: in `_get_prepend_scheme`, return "first" instead of "always" when a`dd_prefix_space=True`. That would fix new conversions to fast ! we just need to handle it in _add_tokens for the cases when loading a `.json` from the hub but should be a bit simpler without new flags 

Thanks for the feedback. Pushed changes.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: xlnet

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45822). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=45822)
**Latest run:** [28796948160:2](https://github.com/huggingface/transformers/actions/runs/28796948160)
**Result:** `failure` | **Jobs:** 15 | **Tests:** 171,542 | **Failures:** 6 | **Duration:** 21h 34m

<!-- ci-dashboard-recap:end -->

Thanks for working on this! 

One concern: `_tokenizer_json_is_byte_level` fetches and fully parses `tokenizer.json` on every `AutoTokenizer.from_pretrained` where `model_type` is `"llama"` - this same file gets fetched and parsed again immediately after by `convert_to_native_format`. .

I'm wondering if we can instead do an existence check instead, like `_has_tekken_tokenizer_file` above it?

Llama1/2 (aentencepiece) always ships `tokenizer.model` I believe and Llama3+ never do 


@itazap thanks, this is the #28218 metaspace prepend_scheme fix, but I can add the byte-level detection here too.

Actually, was your comment meant for https://github.com/huggingface/transformers/pull/47017? I think this can be merged as is.