File size: 256 Bytes
0244572 3dd531b 0244572 3dd531b 0244572 9536ae0 698eb08 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
---
license: mit
tags:
- tokenizers
- transformers.js
- experimental
- auto tokenizer
- llama
- llama 2
- custom code
library_name: transformers
---
# Experimental tokenizer
- Derived from the LLaMa-2 tokenizer
- Inspired by the ChatML & Zephyr tokenizers |