MalvernTok-mini

A 24,000-vocabulary BPE tokenizer designed for SLMs. Uses byte-fallback to handle any Unicode, emojis, or code without producing unknown tokens.

Training Data

Trained on 1.5B tokens from:

from tokenizers import Tokenizer tokenizer = Tokenizer.from_file("tokenizer_24k_bytefallback.json") encoded = tokenizer.encode("Hello, 世界! 🦈")

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support