Add tokenizer.json
#5
by
seanmor5
- opened
We use these models from Elixir which uses the Rust HF tokenizers library which requires a tokenizer.json to load the tokenizer properly
We use these models from Elixir which uses the Rust HF tokenizers library which requires a tokenizer.json to load the tokenizer properly