[LlamaTokenizer] Inconsistent slow vs. fast tokenization when dealing with unknown tokens
Author: xenova
Pull request: no

### System Info

- `transformers` version: 4.36.1
- Platform: Linux-6.2.0-1018-azure-x86_64-with-glibc2.31
- Python version: 3.10.13
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1
- Accelerate version: 0.25.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.1+cu121 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

@ArthurZucker (maybe @Narsil for tokenizers?)

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction


```py
from transformers import AutoTokenizer

text = "\n\t\n"
tokenizer_fast = AutoTokenizer.from_pretrained("RajuKandasamy/tamillama_tiny_30m", use_fast=True)
tokenizer_slow = AutoTokenizer.from_pretrained("RajuKandasamy/tamillama_tiny_30m", use_fast=False)
tokenizer_slow_non_legacy = AutoTokenizer.from_pretrained("RajuKandasamy/tamillama_tiny_30m", use_fast=False, legacy=False)
tokenizer_slow_legacy = AutoTokenizer.from_pretrained("RajuKandasamy/tamillama_tiny_30m", use_fast=False, legacy=True)

print(tokenizer_fast.tokenize(text))               # ['▁', '\n', '<unk>', '\n']
print(tokenizer_fast.encode(text))                 # [1, 31654, 5, 0, 5]
print()
print(tokenizer_slow.tokenize(text))               # ['▁', '\n', '▁', '\n']
print(tokenizer_slow.encode(text))                 # [1, 31654, 5, 31654, 5]
print()
print(tokenizer_slow_non_legacy.tokenize(text))    # ['▁', '\n', '▁', '\n']
print(tokenizer_slow_non_legacy.encode(text))      # [1, 31654, 5, 31654, 5]
print()
print(tokenizer_slow_legacy.tokenize(text))        # ['▁', '\n', '▁', '\n']
print(tokenizer_slow_legacy.encode(text))          # [1, 31654, 5, 31654, 5]
```

### Expected behavior

I'm not quite sure which is the correct behaviour, since this is a custom-built llama tokenizer, and does not include "byte fallback" tokens in the vocabulary. Intuitively, it would make sense to fallback to the unknown token if the byte fallback fails, but I assume we should follow how the sentencepiece implementation does it (which seems to be excluding it?)

Comments:
Another test case:
```py
text = "def main():\n\tpass"
...
# Fast tokenizer
# ['▁def', '▁main', '(', ')', ':', '\n', '<unk>', 'p', 'ass']
# [1, 12849, 17375, 32, 33, 29, 5, 0, 31694, 1917]

# ['▁def', '▁main', '(', ')', ':', '\n', '▁pass']
# [1, 12849, 17375, 32, 33, 29, 5, 4005]

# ['▁def', '▁main', '(', ')', ':', '\n', '▁pass']
# [1, 12849, 17375, 32, 33, 29, 5, 4005]

# ['▁def', '▁main', '(', ')', ':', '\n', '▁pass']
# [1, 12849, 17375, 32, 33, 29, 5, 4005]
```

>  custom-built llama tokenizer

Feels to me we shouldn't care that much for a custom built tokenizer (especially if they removed the byte fallbacks). Byte fallback should definitely try to look for the bytes in the vocab, and resort to using unk if they cannot find them, this is obviously the best choice imo (in terms of logics).

Training byte_fallback without the fallback pieces is the issue I think, not this particular exhibit.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.