akarki15/bdh-nepali

BDH Nepali — Baby Dragon Hatchling

A 25M parameter biologically-inspired language model trained on the NepBERTa Nepali news corpus. Generates Nepali text in Devanagari script using byte-level tokenization (no tokenizer needed — native UTF-8 support).

Try the live demo | GitHub

Architecture

Component	Value
Parameters	25M
Layers	6 (shared weights)
Embedding dim	256
Attention heads	4
Context length	128
Vocab size	256 (byte-level)

Key features:

• Byte-level tokenization — works natively with UTF-8/Devanagari, no tokenizer needed
• Shared weights across layers (iterative refinement)
• Hebbian gating (multiplicative sparse activations)
• ReLU sparsity for interpretable activations
• RoPE position embeddings

Training

• Dataset: NepBERTa corpus — clean Nepali news articles (~52MB)
• Optimizer: AdamW (lr=1e-3, weight_decay=0.1)
• Steps: 10,000
• Hardware: Google Colab T4 GPU (~1 hour)
• Val loss: 0.60 | Perplexity: 1.81

Usage

import torch

model_data = torch.hub.load_state_dict_from_url(
    "https://huggingface.co/akarki15/bdh-nepali/resolve/main/bdh_nepali.pt",
    map_location="cpu"
)
# Load with BDH model class — see GitHub repo for full inference code

Reference

Based on The Dragon Hatchling architecture by Pathway.