Llama tokenizer newline character inconsistency
Author: ingo-m
Pull request: no

- `transformers` version: 4.48.1
- Platform: Linux-6.8.0-51-generic-x86_64-with-glibc2.39
- Python version: 3.12.8
- Huggingface_hub version: 0.27.1
- Safetensors version: 0.5.2
- Accelerate version: 1.3.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.5.1+cu124 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no

This issue relates to tokenizers: @ArthurZucker and @itazap

The Llama 3 documentation says that "Newlines (0x0A) are part of the prompt format", so I guess this is important when tokenizing.

Source:
https://www.llama.com/docs/model-cards-and-prompt-formats/meta-llama-3/#llama-3-instruct

I observed the following:

```python
from transformers import AutoTokenizer

model_id = "meta-llama/Llama-3.2-1B"

tokenizer = AutoTokenizer.from_pretrained(model_id)

# Normal string

newline_token = tokenizer("\n")["input_ids"]
print(newline_token)
# Single newline in normal string:
# [128000, 198]

newline_token = tokenizer("example\nexample")["input_ids"]
print(newline_token)
# Newline with context in normal string:
# [128000, 8858, 198, 8858]

newline_token = tokenizer("\n\n")["input_ids"]
print(newline_token)
# Two newlines in normal string:
# [128000, 271]

newline_token = tokenizer("\\n")["input_ids"]
print(newline_token)
# Double escaped newline in normal string:
# [128000, 1734]

newline_token = tokenizer("example\\nexample")["input_ids"]
print(newline_token)
# Double escapes newline in normal string with context:
# [128000, 8858, 1734, 8858]

# Raw string

newline_token = tokenizer(r"\n")["input_ids"]
print(newline_token)
# Single newline in raw string:
# [128000, 1734]

newline_token = tokenizer(r"\n\n")["input_ids"]
print(newline_token)
# Double newline in raw string:
# [128000, 1734, 1734]

newline_token = tokenizer(r"example\nexample")["input_ids"]
print(newline_token)
# Newline in raw string with context:
# [128000, 8858, 1734, 8858]
```

Now I'm really wondering what's the correct way to represent newlines for a Llama tokenizer.

Using a "normal" string, two consecutive newlines ("\n\n") get represented as 271, vs. "\n" as 198. The prompt template from the website contains double newlines, but it is not clear how they get tokenized.

The raw string r"\n" seems more consistent, the newline gets represented as token 1734 irrespective of context.

So is r"\n" the way the Llama 3 tokenizer expects newlines? Or do the double newlines in the prompt template correspond to token 271?

Any insights into this would be much appreciated. Thanks.

Related (but closed) issue: https://github.com/huggingface/transformers/issues/31030

Comments:
Another related issue: https://github.com/huggingface/transformers.js/issues/1019

Hey! if you want to check which token is associated to which id, you can also use `tokenizer.convert_ids_to_tokens`:
```python
In [9]:  tokenizer.tokenize(r"example\nexample")
Out[9]: ['example', '\\n', 'example']

In [10]:  tokenizer.tokenize("example\nexample")
Out[10]: ['example', 'Ċ', 'example']
```
 and : 
```python
In [5]: tokenizer.convert_ids_to_tokens([128000, 198])
Out[5]: ['
', 'Ċ']

In [6]: tokenizer.convert_ids_to_tokens([128000, 271])
Out[6]: ['
', 'ĊĊ']
```
IMO newline is just `\n` and it is a different token than `\\n`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.