NyayaLM 64K Byte-Level BPE Tokenizer

Fair comparison counterpart to the SentencePiece-BPE NyayaLM-64k tokenizer.

Same corpus (corpus_for_sp.txt)
Same vocab size (64 000)
Same special tokens (Qwen3)
Algorithm: pure GPT-style Byte-Level BPE (Ġ)

Quick start

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("YOUR_HF_REPO")
print(tok.apply_chat_template(
    [{"role": "user", "content": "नेपालको संविधान के हो?"}],
    tokenize=False, add_generation_prompt=True))

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support