LLaMA Tokenizer does not compute word_ids
Author: ikergarcia1996
Pull request: no

### System Info

- `transformers` version: 4.31.0.dev0
- Platform: Linux-4.18.0-477.10.1.el8_8.x86_64-x86_64-with-glibc2.28
- Python version: 3.9.7
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.1.0.dev20230515+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


### Who can help?

@ArthurZucker and @younesbelkada

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The LLaMA tokenizer does not compute word_ids; all the tokens have the id `0`. This function is useful for custom decoding strategies as it allows the user to know to which word in the sentence a token belongs.

```python
from transformers import AutoTokenizer

sentence = "This is a test"

gpt_neox_tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neox-20b")
llama_tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")
gpt2_tokenizer = AutoTokenizer.from_pretrained("gpt2")
t5_tokenizer = AutoTokenizer.from_pretrained("google/mt5-base")
bert_tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

print(gpt_neox_tokenizer([sentence], add_special_tokens=False).word_ids())
print(llama_tokenizer([sentence], add_special_tokens=False).word_ids())
print(gpt2_tokenizer([sentence], add_special_tokens=False).word_ids())
print(t5_tokenizer([sentence], add_special_tokens=False).word_ids())
print(bert_tokenizer([sentence], add_special_tokens=False).word_ids())
```

Output
```python
[0, 1, 2, 3]
[0, 0, 0, 0]
[0, 1, 2, 3]
[0, 1, 2, 2, 3]
[0, 1, 2, 3]
```

### Expected behavior

```python
print(llama_tokenizer([sentence], add_special_tokens=False).word_ids())
```

The expected output is 

```python
[0, 1, 2, 3]
```

Comments:
Hey! Thanks for reporting this. This seems to be related to the backend tokenizer, so most probably the `tokenizers` library. cc @Narsil if you have a quick idea, otherwise, I'll investigate

Oh it's simple.

It's just that what you call "word" doesn't exist for this tokenizer...
A lot of tokenizers use whitespace splitting before processing text, but llama does not.

So the other tokenizers listed here see "This test is" as `"This"+ "test" + "is"` while llama will see exactly "This test is".
So llama thinks its one "word".  So outputting `[0, 0, 0]` is technically correct, even if not too terribly useful.

The best recommendation I can make is using `offsets` instead which will give you the offsets from which each tokens comes from in the original sentence.
This is the only general way that will work on any tokenizer to recover where a given token is from. And you can compute words as you'd like by some other means if you really want to use words (I usually advise to stay away from words, it creates a lot of unecessary issues when looked under enough scrutiny)

```python
from transformers import pipeline, AutoTokenizer

llama_tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-chat-hf")

sentence = "This is a test"

model_inputs = llama_tokenizer(sentence, add_special_tokens=False, return_offsets_mapping=True)

for (input_id, (start, stop)) in zip(model_inputs["input_ids"], model_inputs["offset_mapping"]):
    print(f"{input_id} {start}-{stop} {sentence[start:stop]}")

```


Thank you @Narsil for the great explanation!