🚀 Supernova-Nepali-Tokenizer (Ultra-BPE)

A high-performance, production-ready Byte-Level BPE tokenizer specifically engineered for the Nepali language and Devanagari script. Developed as part of the Supernova project to enable efficient and accurate Nepali LLM processing.

🌟 Key Features

0% Unknown Tokens (UNK): Byte-level fallback ensures every Unicode character (emojis, symbols, rare conjuncts) is representable.
Linguistic Cohesion: Specialized Devanagari Regex pre-tokenizer keeps consonant clusters and matras as atomic units.
Optimized Context Window: Achieves ~3.79 tokens per word, offering a 2.2x compression boost compared to standard GPT-2 tokenizers.
Clean Vocabulary: Saturated at ~2.6k high-frequency tokens for optimized embedding efficiency.

📊 Benchmarks

Tested on the Supernova-teraillm dataset:

Tokenizer	Tokens per Word	Efficiency
Supernova-Nepali (Ultra)	3.79	2.20x Better
GPT-2 (Standard)	8.21	Baseline

🛠️ Usage

Using Transformers

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Supernova11c/Supernova-Nepali-Tokenizer")

text = "नमस्ते, तपाईंलाई कस्तो छ?"
tokens = tokenizer.encode(text)
print(tokenizer.convert_ids_to_tokens(tokens))

🏗️ Architecture

Model: Byte-Level BPE
Vocabulary Size: 2,637
Normalizer: NFC
Pre-tokenizer: ByteLevel + Devanagari Cohesion Regex
Special Tokens: [PAD], [UNK], [BOS], [EOS]

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support