ankushthakurr09/MimansM1_v1

Mimans M1 (514M) - 154.1M Tokens Milestone

Mimans M1 is a 514M parameter decoder-only transformer pretrained from scratch on code, math, and technical text using TileLang custom Blackwell GPU kernels and Muon + AdamW hybrid optimization.

Model Summary

• Parameters: 514,345,526 (~514M)
• Architecture: GQA (10:2), SwiGLU FFN, AttnRes Skip Gating, RMSNorm
• Context Length: 4,096 tokens (max 8,192, RoPE $\theta = 500,000$)
• Vocabulary: 49,152 (Byte-level BPE with FIM support)
• Training Tokens: 154.1M Tokens (Global Step 294)
• Latest Loss: 8.4739
• Hardware: NVIDIA GeForce RTX 5090 (Blackwell sm_120)

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("ankushthakurr09/MimansM1_v1")
tokenizer = AutoTokenizer.from_pretrained("ankushthakurr09/MimansM1_v1")

inputs = tokenizer("def add(a, b):", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0]))