Fix CodeLlama tokenizer stripping real leading spaces on decode (#46491)
Author: kpal002
Pull request: yes

## What this fixes

Closes #46491.

`decoders.Strip(content=" ", left=1)` in `CodeLlamaTokenizer.__init__` was unconditionally removing one leading space on every `decode` call. For inputs that begin with real spaces (e.g. indented Python code), this silently corrupted the output — `decode(encode("    if x:\n"))` returned `"   if x:\n"` instead of `"    if x:\n"`.

The stripping was intended to remove the synthetic `▁` prefix that `Metaspace(prepend_scheme="first")` prepends before the first content token. The bug: after `Replace(▁ → " ")`, the synthetic prefix and real user-provided leading spaces are indistinguishable in the decoded string, so `Strip(left=1)` ate one real space whenever the input started with one or more spaces.

## Root cause

```python
# before (tokenization_code_llama.py line 163-164)
self._tokenizer.decoder = decoders.Sequence(
    [decoders.Replace("▁", " "), decoders.ByteFallback(), decoders.Fuse(), decoders.Strip(content=" ", left=1)]
)
```

## Fix

Remove `Strip` from the decoder sequence and override `_decode` to perform the strip conditionally, based on the first real (non-special) token:

- If the first real token consists **entirely of `▁` characters** (`▁`=29871, `▁▁`=259, `▁▁▁`=1678, …), it was produced by real user-provided leading spaces → **do not strip**.
- Otherwise, the leading space in the decoded text came from the synthetic Metaspace prefix → **strip one space**.

This handles 2, 3, 4, and N leading spaces, and is correct in the presence of a prepended BOS token (BOS is in `all_special_ids` and is skipped when looking for the first real token).

Note: a **single** leading space (`" hello"` vs `"hello"`) is inherently ambiguous — both encode to the same token IDs — and cannot round-trip. This was true before this fix and remains true; it is documented in the new test.

## Before / after

```
# transformers 5.10.x (before)
BUG: '  hello'           -> ' hello'
BUG: '   leading spaces' -> '  leading spaces'
BUG: '    indented_line' -> '   indented_line'

# after this fix
OK:  '  hello'           -> '  hello'
OK:  '   leading spaces' -> '   leading spaces'
OK:  '    indented_line' -> '    indented_line'
OK:  'hello world'       -> 'hello world'   # unaffected
```

## Tests

Added `test_leading_space_roundtrip` to `tests/models/code_llama/test_tokenization_code_llama.py` covering 2-, 3-, and 4-space-prefixed strings via `encode(s, add_special_tokens=False)` → `decode(…, skip_special_tokens=True)`.

Ran locally on this PR's branch (`pytest tests/models/code_llama/test_tokenization_code_llama.py`, Python 3.12.13, transformers 5.10.0.dev0 editable install):
```
60 passed, 8 skipped in 34.11s
```
including `CodeLlamaTokenizationTest::test_leading_space_roundtrip`. Also green on CI: https://circleci.com/gh/huggingface/transformers/2354143

## Disclosure

This fix was drafted with AI-agent assistance (diagnosis, patch, and test). I reviewed every changed line, confirmed the root cause by tracing the `super().__init__()` overwrite of `self._tokenizer` and the BPE merge behavior for multi-▁ tokens, and ran the test suite locally (output above) before opening this PR.

Comments:
cc @arthurzucker @itazap

**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46574)

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: code_llama