Repository: huggingface/transformers (Python)
Date: 2024-05-22 13:01:20+00:00
Task type: Bug Fix

Issue:
`add_prefix_space` won't be respected by Llama tokenizer
### System Info

- `transformers` version: 4.38.2
- Platform: Linux-6.5.0-14-generic-x86_64-with-glibc2.35
- Python version: 3.10.13
- Huggingface_hub version: 0.21.3
- Safetensors version: 0.4.2
- Accelerate version: 0.27.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

With `sentencepiece==0.2.0` and `protobuf==4.25.3` installed


### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf", local_files_only=True, add_prefix_space=False)
>>> tokenizer.tokenize("overheard")
['▁over', 'he', 'ard']
```

Also tried `add_dummy_prefix_space=False`, the output is still the same.

### Expected behavior

The tokenize result should not add prefix space (`SPIECE_UNDERLINE`)


Issue discussion:
Hey, I took a peek under the hood and looks like setting `add_prefix_true` is only changing `kwargs[slow]=True` (in [tokenization_llama_fast.py](https://github.com/huggingface/transformers/blob/5011908e10d9592eeb634f4940e0bc130d3edc69/src/transformers/models/llama/tokenization_llama_fast.py#L127C9-L132C1). The `super().__init__()` method should receive this parameter if set.
Passing this in seems to work in preliminary tests
```python
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf", add_prefix_space=False)
>>> tokenizer.tokenize('overheard')
['over', 'he', 'ard']
```

Mind if I take this up @ArthurZucker & @scruel?

Edit: For completeness, showing that behavior is unchanged when `add_prefix_space=True`
```
>>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf", add_prefix_space=True)
>>> tokenizer.tokenize('overheard')
>>> ['\u2581over', 'he', 'ard']
```
You always can take by creating a PR.
Thank you, made a pull request. This was happening in `T5TokenizerFast` as well.
Thanks I'll review asap!
closing as #28881 fixed it! 
@ArthurZucker are you sure this is fixed? I am still experiencing this in 4.41.0:
![image](https://github.com/huggingface/transformers/assets/1677826/aa907c51-f6d0-4154-a917-f13ba4c840f3)

I can also still not see it being used here:
https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/tokenization_llama_fast.py#L153
You need to se `from_slow=True` to trigger conversion
It is used in `convert_slow` 😉 
This is very confusing and not transparent to the user at all.
If I just use the `AutoTokenizer` class with default settings I would expect this to work and not silently do nothing.
It should at least give a warning, or rather set the `from_slow` then automatically.
I agree with you, on main there is this:
```python
        if add_prefix_space is not None:
            logger.warning_once(
                "You set `add_prefix_space`. The tokenizer needs to be converted from the slow tokenizers"
            )
            kwargs["from_slow"] = True
```
which should give you a warning and automatically convert it 
But it does not seem to be taken into account. @itazap would be nice if you can investigate and open a PR to make sure it forces from flow:
```python3
In [1]: from transformers import AutoTokenizer
tokenizer
In [2]: tokenizer = AutoTokenizer.from_pretrained("meta-llama/llama-2-7b-hf",add_prefix_space=False)
You set `add_prefix_space`. The tokenizer needs to be converted from the slow tokenizers
You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565

In [3]: tokenizer.encode("Hey")
Out[3]: [1, 18637]

In [4]: tokenizer.tokenize("Hey")
Out[4]: ['▁Hey']

In [5]: tokenizer = AutoTokenizer.from_pretrained("meta-llama/llama-2-7b-hf",add_prefix_space=False, from_slow=True)

In [6]: tokenizer.tokenize("Hey")
Out[6]: ['H', 'ey']

In [7]: tokenizer = AutoTokenizer.from_pretrained("meta-llama/llama-2-7b-hf",add_prefix_space=False)

In [8]: tokenizer.tokenize("Hey")
Out[8]: ['▁Hey']
```
^^ Thanks

Another thing I noted, is that if I specify `from_slow` in `tokenizer_config.json` then it is ignored. Is this expected behavior?

Solution patch:
diff --git a/src/transformers/models/llama/tokenization_llama_fast.py b/src/transformers/models/llama/tokenization_llama_fast.py
--- a/src/transformers/models/llama/tokenization_llama_fast.py
+++ b/src/transformers/models/llama/tokenization_llama_fast.py
@@ -163,6 +163,7 @@ def __init__(
             add_bos_token=add_bos_token,
             add_eos_token=add_eos_token,
             use_default_system_prompt=use_default_system_prompt,
+            add_prefix_space=add_prefix_space,
             legacy=legacy,
             **kwargs,
         )


Test patch:
diff --git a/tests/models/llama/test_tokenization_llama.py b/tests/models/llama/test_tokenization_llama.py
--- a/tests/models/llama/test_tokenization_llama.py
+++ b/tests/models/llama/test_tokenization_llama.py
@@ -602,6 +602,10 @@ def test_special_token_special_word(self):
         self.assertEqual(decoded_tokens, "hello")
 
     def test_no_prefix_space(self):
+        tokenizer_no_prefix_space = LlamaTokenizerFast.from_pretrained("huggyllama/llama-7b", add_prefix_space=False)
+        no_prefix_space_tokens = tokenizer_no_prefix_space.tokenize("Hey")
+        self.assertEqual(no_prefix_space_tokens, ["H", "ey"])
+
         tokenizer = LlamaTokenizerFast.from_pretrained(
             "huggyllama/llama-7b", legacy=False, from_slow=True, add_prefix_space=False
         )