Viu-Mini-MoE — 242M total / 166M active (28 layers)

Hinglish-first research MoE. Pehle batata hu, tum approve karoge, tabhi code chhedege — ye repo isi rule se bana hai.

Model details

Arch: VIU-1 Frontier MoE — 28 layers (19 dense + 9 micro-MoE), dim 640, H=10, KV=2, vocab 48k, tied embeddings
Attention: GQA + RMSNorm + QK-Norm + RoPE theta 500k (YaRN-ready), hybrid Sliding-512 + Full-2048 (ratio 3:1)
FFN: SwiGLU dense + micro-MoE (8 experts @ 1/4 size, top-2 + 1 shared, balance loss 0.01)
Aux: Multi-Token (next-next) head
Params verified: model/scripts/viu1_moe.py smoke — total 241.7M / active 166.2M
Context: 2048 (YaRN se 8k tak)
Training: model/scripts/train.py (AdamW + cosine 5% warmup + grad-clip 1.0 + BF16 + grad-accum)

Structure (modular)

Mini-ViuAI-50M/  <- ye repo root (HF model repo ke liye)
  model/
    configs/viu1_moe_config.yaml + config.json
    scripts/viu1_moe.py, train.py, push_to_hf.py
    checkpoints/  (LFS)
  tokenizer/
    configs/tokenizer_config.yaml
    outputs/tokenizer.json (FINAL 48k, abhi TEST 36252 bytes)
    scripts/tokenizer_train.py
  data/
    raw/ hinglish.txt (40%) hindi.txt (30%) english.txt (30%)  -> HF dataset repo me jayega
    eval/sample_test.txt
  docs/
    PROGRESS.md, VIU1_MOE_PLAN.md, HF_PUSH_GUIDE.md

Tokenizer

48k byte-level BPE, NFC + ByteLevel. Mix 40:30:30.

# local 500k lines
python tokenizer/scripts/tokenizer_train.py --data_dir ./data/raw --out ./tokenizer/outputs/tokenizer.json --vocab_size 48000 --max_lines 500000
# HF dataset se (ready ke baad)
python tokenizer/scripts/tokenizer_train.py --hf_dataset ViuAI/viu-mini-pretrain-40-30-30 --out ./tokenizer/outputs/tokenizer.json

Training

pip install -r requirements.txt
# smoke (CPU, 3 steps, tiny model — loop test)
python model/scripts/train.py --smoke

# real (needs CUDA + FINAL tokenizer + HF dataset)
python model/scripts/train.py --config model/configs/train_config.yaml --model_config model/configs/viu1_moe_config.yaml
# ya HF dataset streaming
python model/scripts/train.py --hf_dataset ViuAI/viu-mini-pretrain-40-30-30 --config model/configs/train_config.yaml

HF push

See docs/HF_PUSH_GUIDE.md. Model repo: ViuAI/ViuMini-MoE-242M, Dataset repo: ViuAI/viu-mini-pretrain-40-30-30. Requires huggingface_hub login — token .hf_token me nahi, hf auth login se.

Status

Code ready, smoke pass (loss 6.18→6.01, 28L 241M smoke OK)
Pending: real 4-5B tokens → FINAL 48k tokenizer → CUDA training → SFT/DPO
Notes: har folder me NOTES.md, daily docs/PROGRESS.md

Disclaimer

Research purpose. MoE fail risk 50-60% first run, 85% after retune — prevention checklist docs/VIU1_MOE_PLAN.md me.

Downloads last month: -; Downloads are not tracked for this model. How to track

ViuAI
/

ViuMini-MoE-242M