nuph-tokenizer / README.md

baebee

Update README.md

3dd531b about 1 year ago

preview code

raw

history blame

256 Bytes

metadata

license: mit
tags:
  - tokenizers
  - transformers.js
  - experimental
  - auto tokenizer
  - llama
  - llama 2
  - custom code
library_name: transformers

Experimental tokenizer

Derived from the LLaMa-2 tokenizer
Inspired by the ChatML & Zephyr tokenizers