Supernova11c/Supernova-NepaliFast-V6

Supernova NepaliFast V5 — Rust Native Tokenizer Engine

This directory contains the completed, optimized Rust Native Tokenizer Engine (supernova_rust) for Supernova NepaliFast V5. It is built using PyO3 and Maturin, delivering 15M+ characters per second (running 3.86x faster than OpenAI's highly optimized tiktoken engine).

🚀 Performance Overview

• Supernova Rust Engine: 15.2M chars/sec (1.02 ms average on sample document)
• OpenAI Tiktoken (cl100k_base): 3.9M chars/sec (3.96 ms average on sample document)
• Speedup: 3.86x faster than tiktoken

---

🛠️ Build and Compilation Guide

Follow these steps to compile and install the high-performance native module directly from this source code.

1. Prerequisites

Install Cargo, Rustc, and Maturin:

sudo apt-get update && sudo apt-get install -y cargo
pip install maturin

2. Compile and Install in Release Mode

Run this inside the supernova_rust directory containing Cargo.toml:

maturin build --release

After building, install the newly created .whl package using pip:

pip install --force-reinstall target/wheels/*.whl

---

💻 Quickstart Usage (Python)

Once compiled and installed, you can use the Rust Engine in your Python workspace as follows:

import supernova_rust

# Initialize the native engine using Python vocabularies
rust_engine = supernova_rust.SupernovaV5Engine(python_main_vocab, python_unicode_vocab)

# Encode text
tokens = rust_engine.encode("नमस्ते! Supernova AI 🚀")
print("Encoded Tokens:", tokens)

# Decode tokens
decoded_text = rust_engine.decode(tokens)
print("Decoded Text:", decoded_text)