CodeLlama tokenizer strips one leading space on encode→decode round-trip (regression vs v4)
Author: SuryanshSS1011
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 5.10.2
- `tokenizers` version: 0.22.2
- Python version: 3.11.2
- Platform: Linux (Penn State HPC, RHEL 8.10)
- PyTorch version: not relevant (tokenizer-only repro)

### Who can help?

@ArthurZucker @itazap

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

# CodeLlama tokenizer drops one leading space on decode round-trip (v5 regression)
 
## Reproduction
 
```python
from transformers import AutoTokenizer
 
tok = AutoTokenizer.from_pretrained("codellama/CodeLlama-7b-hf")
for s in ["   leading spaces", "    indented_line"]:
    decoded = tok.decode(tok.encode(s, add_special_tokens=False))
    print(f"{'OK' if decoded == s else 'BUG'}: {s!r} -> {decoded!r}")
```
 
Output on `transformers 5.10.2` + `tokenizers 0.22.2`:
 
```
BUG: '   leading spaces' -> '  leading spaces'
BUG: '    indented_line' -> '   indented_line'
```
 
The same script on `transformers 4.55.4` (last v4) round-trips correctly:
 
```
OK: '   leading spaces' -> '   leading spaces'
OK: '    indented_line' -> '    indented_line'
```
 
Exactly one leading space is lost on round-trip; trailing whitespace, internal whitespace, and other characters round-trip correctly.
 
## Scope
 
All CodeLlama variants tested are affected:
 
| Checkpoint | Class | Round-trips |
|---|---|---|
| `codellama/CodeLlama-7b-hf` | `CodeLlamaTokenizer` | 1/4 |
| `codellama/CodeLlama-7b-Instruct-hf` | `CodeLlamaTokenizer` | 1/4 |
| `codellama/CodeLlama-7b-Python-hf` | `CodeLlamaTokenizer` | 1/4 |
| `codellama/CodeLlama-13b-hf` | `CodeLlamaTokenizer` | 1/4 |
| `codellama/CodeLlama-34b-hf` | `CodeLlamaTokenizer` | 1/4 |
| `codellama/CodeLlama-70b-hf` | `LlamaTokenizer` | 1/4 |
 
## Root cause
 
`src/transformers/models/code_llama/tokenization_code_llama.py`, lines 163–164:
 
```python
self._tokenizer.decoder = decoders.Sequence(
    [decoders.Replace("▁", " "), decoders.ByteFallback(), decoders.Fuse(), decoders.Strip(content=" ", left=1)]
)
```
 
The trailing `decoders.Strip(content=" ", left=1)` unconditionally strips one leading space on decode. This was intended to remove the synthetic SentencePiece `▁` prefix introduced by the normalizer (`Prepend ▁` → `Replace " " → ▁`). But it also eats one leading space when the user's input itself starts with a space, because by that point in the decode pipeline the synthetic prefix and user-provided leading spaces are indistinguishable.
 
v4's `CodeLlamaTokenizerFast` had different decode handling that distinguished these. The v5 migration that removed the Fast class (#40936 "rm slow tokenizers", #42563 "Refactor-tokenization-more", #42894 "use TokenizersBackend") landed this Strip-step decoder for all CodeLlama checkpoints.
 
## Impact
 
This silently corrupts whitespace-sensitive workloads on CodeLlama. In code generation specifically, where indentation is semantically meaningful:
 
```python
input  = "    if x > 0:\n        return x"
output = tok.decode(tok.encode(input, add_special_tokens=False))
# output = "   if x > 0:\n        return x"   # 3 spaces instead of 4 — silent indentation corruption
```
 
Any pipeline using `encode → decode` for round-trip text comparison (BLEU/chrF/exact-match metrics, RL reward signals on code tokens, fine-tuning data validation) produces silently-wrong values when leading whitespace is present.
 
## Suggested fix direction
 
Two possible shapes; deferring to maintainers:
 
1. **Track the synthetic prefix.** Instead of `Strip(left=1)`, the decoder should only strip a leading space when the corresponding encoded sequence began with the synthetic `▁` prefix and *not* with a user-provided leading space. This requires the decoder to know whether `add_prefix_space` would have fired.
2. **Replace with v4-style decode logic.** Port whatever `CodeLlamaTokenizerFast.decode` did in v4 (which round-tripped correctly) into the current implementation.
Happy to implement once a direction is preferred.

### Expected behavior

`tok.decode(tok.encode(s))` should round-trip leading whitespace, as it does on transformers v4 and on most other tokenizers in v5 (Qwen, StarCoder2, Phi-3 all round-trip correctly on the same test strings).
 
This is a tokenization-correctness bug. Code generation and any whitespace-sensitive evaluation pipeline silently produce wrong values on CodeLlama checkpoints.

Comments:
Hi, I'd like to work on this issue. Before I start implementing, could the maintainers clarify which fix direction is preferred?

1. Track the synthetic prefix  only strip the leading space when it was introduced by the normalizer, not when the user's input 2. starts with a space
Port the v4-style CodeLlamaTokenizerFast decode logic into the current implementation

Happy to implement once a direction is confirmed. Thanks!

Investigated the fix paths further. The minimal "drop the `Strip`" approach is incomplete as it surfaces a different round-trip bug. Trace below.

## Token-ID ambiguity

On current main with `CodeLlama-7b-hf`:

| Input | Token IDs | Tokens |
|---|---|---|
| `"hello world"` | `[22172, 3186]` | `['▁hello', '▁world']` |
| `" hello world"` | `[22172, 3186]` | `['▁hello', '▁world']` |
| `"  hello"` | `[29871, 22172]` | `['▁', '▁hello']` |

`"hello world"` and `" hello world"` encode to **identical IDs**, so the single-leading-space case is genuinely ambiguous after encode and no decoder can distinguish them.

The current `Strip(content=" ", left=1)` chooses the no-leading-space interpretation, which is what users coding against LLMs usually want (`decode(encode("hello"))` should return `"hello"`).

`"  hello"` (two leading spaces) is the *distinguishable* case: `'▁'` appears as a standalone token. `Strip` over-strips here, removing one of two real spaces.

## Effect of just removing `Strip`

```python
tok.backend_tokenizer.decoder = decoders.Sequence(
    [decoders.Replace("▁", " "), decoders.ByteFallback(), decoders.Fuse()]
)
```

| Input | Before fix | After dropping Strip |
|---|---|---|
| `"hello world"` | `"hello world"` ✅ | `" hello world"` ❌ |
| `" hello world"` | `"hello world"` ❌ | `" hello world"` ✅ |
| `"  hello"` | `" hello"` ❌ | `"  hello"` ✅ |
| `"x  hello"` | `"x  hello"` ✅ | `" x  hello"` ❌ |

So it just trades one over-strip bug for an over-keep bug. The current behavior is a deliberate choice in the ambiguous case; the actual bug is that `Strip` is too aggressive when there's a standalone `'▁'` token at position 0.

## Proposed fix

Replace the unconditional `Strip` with a conditional one that only strips when the leading character of the decoded result wasn't produced by a standalone `'▁'` token. Two implementation paths:

1. **Custom decode wrapper in `CodeLlamaTokenizer` overriding `_decode`:** peek at `token_ids[0]` and if it equals the ID for standalone `'▁'` (typically `29871`), skip the leading-space strip. Cleanest from a tokenizers-rs perspective, no new Rust decoder needed.
2. **Drop `Strip` from the decoder `Sequence` and handle the prepend at encode-time instead** so decode is unambiguous. More invasive, would touch the `Metaspace` pre-tokenizer config.

Path 1 is the smaller change and stays inside Python. Happy to PR that.

@ArthurZucker @itazap, any preference on direction, or reason to prefer keeping the current over-strip behavior?

Opened a fix in #46574.

## What the fix does

The root cause identified in the issue is correct: `decoders.Strip(content=" ", left=1)` unconditionally removes one leading space on every decode call, conflating the synthetic Metaspace prefix with real user-provided leading spaces.

**One subtlety beyond the issue description:** for inputs with 3+ leading spaces, the BPE does not always produce a standalone `▁` (29871) as the first token. It can produce multi-`▁` merged tokens instead:

| Input | First token | Token text |
|-------|-------------|------------|
| `"  hello"` (2 spaces) | 29871 | `▁` |
| `"   leading"` (3 spaces) | 259 | `▁▁` |
| `"    indent"` (4 spaces) | 1678 | `▁▁▁` |

So checking `token_ids[0] == 29871` (Path 1 as originally described) fixes the 2-space case but leaves the 3- and 4-space cases broken. The correct predicate is: **the first real (non-special) token's text consists entirely of `▁` characters**.

## The fix (two changes)

**1. Remove `Strip` from the decoder sequence** (`tokenization_code_llama.py`):

```python
# before
self._tokenizer.decoder = decoders.Sequence(
    [decoders.Replace("▁", " "), decoders.ByteFallback(), decoders.Fuse(), decoders.Strip(content=" ", left=1)]
)

# after
self._tokenizer.decoder = decoders.Sequence(
    [decoders.Replace("▁", " "), decoders.ByteFallback(), decoders.Fuse()]
)
```

**2. Override `_decode` with a conditional strip:**

```python
def _decode(self, token_ids, skip_special_tokens=False, **kwargs):
    text = super()._decode(token_ids, skip_special_tokens=skip_special_tokens, **kwargs)

    if not self.add_prefix_space or not text.startswith(" "):
        return text

    # The Metaspace prepend gives a synthetic ▁ that merges INTO the first
    # content token (▁hello = 22172), so its space is not distinguishable
    # and must be stripped. Real leading spaces produce one or more pure-▁
    # tokens (▁=29871, ▁▁=259, ▁▁▁=1678, …) ahead of the first content
    # token. A pure-▁ token is one whose text is entirely ▁ characters.
    # Skip any leading BOS/special tokens before inspecting the first real token.
    all_special_ids = set(self.all_special_ids)
    first_real_id = next((t for t in token_ids if t not in all_special_ids), None)
    if first_real_id is not None:
        first_real_token = self.convert_ids_to_tokens(first_real_id) or ""
        if all(c == SPIECE_UNDERLINE for c in first_real_token):
            return text  # real leading space(s) — preserve them

    return text[1:]  # synthetic prefix — strip it
```

The BOS check (`all_special_ids`) handles the default `add_bos_token=True` case correctly without needing separate logic.

## Verified behaviour after fix

```
OK: 'hello world'       -> 'hello world'
OK: '  hello'           -> '  hello'
OK: '   leading spaces' -> '   leading spaces'
OK: '    indented_line' -> '    indented_line'
OK: 'x  hello'          -> 'x  hello'
```

The single-leading-space case (`" hello"` vs `"hello"`) remains ambiguous — both encode to identical IDs — and is unchanged from before.

Thanks for picking this up. The pure-▁-token predicate is a real improvement over what I had in mind. I was checking token_ids[0] == 29871 directly, which fixes 2 leading spaces but silently breaks for 3+ where BPE merges to 259 (▁▁) or 1678 (▁▁▁). Walking the token text and confirming every char is ▁ covers all those cases in one line.

One small thought: a single negative-test case asserting that decode(encode("hello world")) is still "hello world" (i.e. the synthetic prefix path still strips correctly) would lock in the other direction of the bug. The current test covers the preserve-leading-spaces direction but not the original "synthetic prefix gets stripped" behaviour you're keeping.

Added.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.