`LlamaTokenizerFast` wrong `word_id` references based on batch encoding
Author: vasqu
Pull request: no

### System Info

- `transformers` version: 4.39.0.dev0 (commit hash https://github.com/huggingface/transformers/commit/923733c22bf4d3cc6661c8cd3b730b275e9a938e)
- Platform: Linux-6.5.0-10022-tuxedo-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.2.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer

# any llama (1) based tokenizer
llama_tokenizer = AutoTokenizer.from_pretrained('hf-internal-testing/llama-tokenizer')

sentence = 'Sampel to demonstrate the issue'

input_encoding = llama_tokenizer(sentence, add_special_tokens=True)
tokens = input_encoding.tokens()
word_ids = input_encoding.word_ids()

print(f'Tokens produced: {tokens}\nReferenced word ids: {word_ids}')
```
Prints: 
`Tokens produced: ['
', '▁S', 'amp', 'el', '▁to', '▁demonstrate', '▁the', '▁issue']`
`Referenced word ids: [None, 0, 0, 0, 0, 0, 0, 0]`

### Expected behavior

Given a fast tokenizer, batch encodings should be able to return the tokenized tokens as well as the corresponding `word_ids`. The first `None` is to be expected though (see this [example](https://github.com/huggingface/transformers/issues/12665#issuecomment-879161405)). 

Comments:
Alright this is a valid bug! Pretty sure this come from the lack of a `pre_tokenizer` for llama and shall thus be fixed by #28881

Here is what I am getting with the `tokenizers` [PR](https://github.com/huggingface/tokenizers/pull/1476/files) of course 
```python
In [1]: from transformers import AutoTokenizer
   ...: 
   ...: # any llama (1) based tokenizer
   ...: llama_tokenizer = AutoTokenizer.from_pretrained('hf-internal-testing/llama-tokenizer',
   ...: from_slow=True, legacy=False)
   ...: 
   ...: sentence = 'Sample to demonstrate the issue'
   ...: 
   ...: input_encoding = llama_tokenizer(sentence, add_special_tokens=True)
   ...: tokens = input_encoding.tokens()
   ...: word_ids = input_encoding.word_ids()
   ...: 
   ...: print(f'Tokens produced: {tokens}\nReferenced word ids: {word_ids}')
Tokens produced: ['
', '▁S', 'amp', 'el', '▁to', '▁demonstrate', '▁the', '▁issue']
Referenced word ids: [None, 0, 0, 0, 1, 2, 3, 4]
```


Thank you, this is looking good!

Hi @ArthurZucker, I am still facing the same issue even tried your code.

I tried to use your PD, but it seems to be merged to the main. 

You need to make sure you are using the correct `tokenizers` version. Note that you should also disable `split` for `Metaspace`


> You need to make sure you are using the correct `tokenizers` version. Note that you should also disable `split` for `Metaspace`

I am using `tokenizers` version `0.16.0-dev.0`. Is this the correct version?

Besides, could you please how to disable `split` for `Metaspace`?

It is the correct version. Setting `tokenizer._tokenizer.pre_tokenizer.split = False` 

```python
from tokenizers import Tokenizer
llama_tokenizer = Tokenizer.from_pretrained('meta-llama/Llama-2-7b-hf')
llama_tokenizer._tokenizer.pre_tokenizer.split = False
``` 

This raises error: `AttributeError: 'tokenizers.Tokenizer' object has no attribute '_tokenizer'. Did you mean: 'pre_tokenizer'?`. And if I replace the row with `llama_tokenizer.pre_tokenizer.split`, it returns `AttributeError: 'NoneType' object has no attribute 'split'`

```python
from tokenizers import Tokenizer
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained('meta-llama/Llama-2-7b-hf', from_slow=True, legacy=False)
tok._tokenizer.pre_tokenizer.split = False
```

This raises error: `AttributeError: 'NoneType' object has no attribute 'split'`

Could you please help?

Before I start, I use the most recent tokenizers version in main branch and the transformers version in the PR #28881(with a slight modification locally on my end)


Hey @phusroyal, I've just checked it myself again and you don't need to set `split=False`. In my case, it would cause unwanted behaviour. In the `tokenizers` library, if you need it in the future, you would access it via `tokenizer.pre_tokenizer.split = False` I think. 

```python
from tokenizers import Tokenizer

# any llama (1) based tokenizer
llama_tokenizer = Tokenizer.from_pretrained('hf-internal-testing/llama-new-metaspace')
# just for demonstration purpose, it's also here
#llama_tokenizer.pre_tokenizer.split = False

sentence = ['Sampel to demonstrate the issue']

input_encoding = llama_tokenizer.encode_batch(sentence, add_special_tokens=True)
tokens = input_encoding[0].tokens
word_ids = input_encoding[0].word_ids

print(f'Tokens produced: {tokens}\nReferenced word ids: {word_ids}')
```

For the transformers variant, you would also not want to modify the `split` attribute (albeit you'd do it like `tokenizer._tokenizer.pre_tokenizer.split = False` like @ArthurZucker showed). Although, I encountered an issue when initializing the tokenizer: `add_prefix_space` wasn't recognized as a keywordarg anymore (see [this line](https://github.com/huggingface/transformers/blob/03732dea60fba1da78c79eb59c443ebf975c2be6/src/transformers/convert_slow_tokenizer.py#L604)). Not sure if it's because I'm messing in dev versions or if the metaspace refactor caused some issues there.

Ah and another small thing, this issue is related to llama1 tokenizers. Not sure how much it relates to llama2 tokenizers. 

Great! thank you for your help. 

And yes, I also face the same issue in llama2 tokenizers.

I see. I don't think llama2 has a pretokenizer in the current implementation of Tokenizers. But, when testing it with `meta-llama/Llama-2-7b-hf`, the same transformers settings worked for me, i.e. `AutoTokenizer.from_pretrained('meta-llama/Llama-2-7b-hf', from_slow=True, legacy=False)`. 

Regarding tokenizers, it seems the pretokenizer isn't set in that case which causes errors. Might be a separate bug/issue.

The Llama2 and 1 don't have a pre_tokenizer, but there are issues with the normalizer, thus the fix 


I'll get the PR ready in a bit

I ended up here from a separate problem with Gemma and pre-tokenizers not working at all.
I really appreciate all the hard work hf is putting into transformers, we wouldn't be here without you and thanks for it. 

But this codebase has so much debt, and it is so incredibly messy, guys...

Could you share a reproducer? 
Gemma tokenizer is compeltely different

@ArthurZucker, it's the same issue as here with llama. Since it's gated I've used the `huggingface-cli login` beforehand. Transformers version is the most recent one `4.40.0`.

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('google/gemma-7b', use_auth=True)
sentence = 'Sampel to demonstrate the issue'

input_encoding = tokenizer(sentence, add_special_tokens=True)
tokens = input_encoding.tokens()
word_ids = input_encoding.word_ids()

print(f'Tokens produced: {tokens}\nReferenced word ids: {word_ids}')
```
Results in:
`Tokens produced: ['<bos>', 'Samp', 'el', '▁to', '▁demonstrate', '▁the', '▁issue']`
`Referenced word ids: [None, 0, 0, 0, 0, 0, 0]`

Also tried with the versions above (i.e. the PR #28881 branch + the tokenizers `0.16.0-dev.0` version) using `legacy=False` and `from_slow=True` to no avail. I know it's not meant to fix it, just gave it a try.

Yep, it's the same issue here, the `normalizer` does not split the input on `spaces` like the `pre_tokenizer` does. Will see what I can do, basically this should work:
```python3
>>> from tokenizers import pre_tokenizers, normalizers
>>> tokenizer._tokenizer.normalizer = normalizers.Sequence([])
>>> tokenizer._tokenizer.pre_tokenizer = pre_tokenizers.Metaspace(prepend_scheme="never", split=True)
>>> input_encoding = tokenizer(sentence, add_special_tokens=True)
>>> input_encoding.tokens()
['<bos>', 'Samp', 'el', '▁to', '▁demonstrate', '▁the', '▁issue']
input_encoding.word_ids()
```


This was not really possible before without adding extra bugs, but should work now 

Also now that `tokenizers` has been updated I'll officially fix the llama tokenizer

Thank you :pray: is there another separate issue open for gemma or should I (or someone else) open a separate issue?

Mmm feel free to open one, I'll fix it in the same PR !



Hi all, 

It seems this issue was fixed so perhaps I'm doing something wrong, but the issue persists whenever I try to retrieve the word_ids for a sequence with the Llama tokenizer. For example below,

```
tokenizer = AutoTokenizer.from_pretrained('meta-llama/Llama-2-7b-hf') 
tokenizer.add_special_tokens({'pad_token': '[PAD]'})
tokenizer.pad_token = tokenizer.eos_token
encoded = tokenizer("Hi my name is Adam")
print(encoded.tokens)
print("WORD IDS", encoded.word_ids())
```
still returns the following 
```
['
', '▁Hi', '▁my', '▁name', '▁is', '▁Adam']
WORD IDS [None, 0, 0, 0, 0, 0]
```

I deleted and redownloaded the `transformers` package as well and my versions from `pip list` look like below. Is anyone still able to replicate this issue? 

```
tokenizers               0.19.1
torch                    2.0.1
transformers             4.43.3
```

Hey! You are right, it is not fixed but I recommend you to use:
```python3

In [30]: tokenizer("Hi my name is Adam").encodings[0].offsets
do_tokenizer None
Out[30]: [(0, 0), (0, 2), (2, 5), (5, 10), (10, 13), (13, 18)]
```
because the concept of `word` is very arbitrary. The main reason why it works for gpt2 for example is because the `Split` pre tokenizer splits the string into "words". But since we are not doing that, only merging what is mergeable.
For example:
```
print(tokenizer("Hi my name is   Adam. we'll").encodings[0].tokens)
['
', '▁Hi', '▁my', '▁name', '▁is', '▁▁', '▁Adam', '.', '▁we', "'", 'll']
```
The `'▁▁'` is a word? Or does it belong to `'▁▁'` by itself? What about `'` and `ll` ? 

I recommend you to use `pre_tokenizers.Split(Regex('(?<!▁)▁'), "merged_with_next")` for example: 
```python3
tokenizer._tokenizer.pre_tokenizer = pre_tokenizers.Split(Regex('(?<!▁)▁'), "merged_with_next") (on top of metaspace)
```