GemmaTokenizer - unclear purpose of Split pre_tokenizer on whitespace
Author: biuq
Pull request: no



In [GemmaTokenizer](https://github.com/huggingface/transformers/blob/a7f29523361b2cc12e51c1f5133d95f122f6f45c/src/transformers/models/gemma/tokenization_gemma.py#L28) there is a `Split` `pre-tokenizer` that splits the text on whitespace with `merged_with_previous` split behavior.

https://github.com/huggingface/transformers/blob/a7f29523361b2cc12e51c1f5133d95f122f6f45c/src/transformers/models/gemma/tokenization_gemma.py#L98

However, what effect does it have if `normalizer` runs first that replaces whitespace characters with "▁" character?

https://github.com/huggingface/transformers/blob/a7f29523361b2cc12e51c1f5133d95f122f6f45c/src/transformers/models/gemma/tokenization_gemma.py#L97

Comments:
opening a PR shortly to fix this.

I investigated this on current main and confirmed that the GemmaTokenizer whitespace pre-tokenizer is redundant.

GemmaTokenizer normalizes literal spaces (" ") to the SentencePiece underline character ("▁"). As a result, a pre_tokenizer=Split(" ", "merged_with_previous") is a no-op, since no literal spaces remain after normalization.

How I tested:
	•	Verified via backend_tokenizer introspection that normalization runs before any pre-tokenization.
	•	Disabled the pre-tokenizer at runtime and compared tokenize() and encode() outputs against the baseline.
	•	Tested with multiple whitespace patterns (multiple spaces, tabs, mixed whitespace).
	•	Observed no differences in tokens or token IDs.

I’ve opened a PR that removes the redundant pre-tokenizer, cleans up the unused import, and adds a regression test to ensure tokenization behavior remains unchanged going forward.

https://github.com/huggingface/transformers/pull/43106
@biuq  please take a look at it.

I used the following test script to validate the redundancy.

```
from transformers import AutoTokenizer

def get_backend(tok):
    if hasattr(tok, "backend_tokenizer"):
        return tok.backend_tokenizer
    if hasattr(tok, "_tokenizer"):
        return tok._tokenizer
    raise RuntimeError(f"Cannot find backend tokenizer on {type(tok)}")

def main():
    tok = AutoTokenizer.from_pretrained("hf-internal-testing/dummy-gemma", use_fast=True)
    bt = get_backend(tok)

    print("Tokenizer class:", type(tok))
    print("Normalizer:", bt.normalizer)
    print("PreTokenizer:", bt.pre_tokenizer)

    texts = [
        "Hello   this\tis  a test",
        "Hello this is a test",
        " Hello this is a test",
        "Hello\tthis is a test",
        "Hello\nthis is a test",
        "Hello\u00A0this is a test", 
        "a  b   c",
        "a\tb c",
        "a b\tc",
    ]

    # baseline
    baseline_tok = [tok.tokenize(t) for t in texts]
    baseline_ids = [tok.encode(t) for t in texts]

    # disable pretokenizer
    orig = bt.pre_tokenizer
    bt.pre_tokenizer = None

    no_pre_tok = [tok.tokenize(t) for t in texts]
    no_pre_ids = [tok.encode(t) for t in texts]

    # restore
    bt.pre_tokenizer = orig

    print("\n== Results ==")
    for i, t in enumerate(texts):
        same_ids = baseline_ids[i] == no_pre_ids[i]
        same_tok = baseline_tok[i] == no_pre_tok[i]
        if not (same_ids and same_tok):
            print("\nDIFF for:", repr(t))
            print("tokenize baseline:", baseline_tok[i])
            print("tokenize no_pre :", no_pre_tok[i])
            print("ids baseline:", baseline_ids[i])
            print("ids no_pre :", no_pre_ids[i])
            break
    else:
        print("No differences found: disabling pre_tokenizer does not change tokenize() or encode() for all test strings.")

if __name__ == "__main__":
    main()
```