ULTRON-mini-512 Foundation Model (Generation 1)

ULTRON-mini-512 is a compact foundation language model built entirely from scratch in PyTorch.

Architecture Specifications

• Parameters: 520,173,056 (~512M)
• Hidden Size ($d_{model}$): 1024 ($2^{10}$)
• Layers ($L$): 36
• Attention Heads ($n_q / n_{kv}$): 16 / 8 (Grouped-Query Attention 2:1)
• Intermediate Size ($d_{ff}$): 3072 (SwiGLU)
• Vocabulary Size ($V$): 32,768 ($2^{15}$ Byte-Level BPE)
• Context Length ($T$): 2048 ($2^{11}$)
• Positional Embeddings: Rotary Position Embeddings (RoPE, $\theta = 500,000$)
• Normalization: FP32-stabilized RMSNorm with QK-LayerNorm
• Training Precision: Mixed Precision FP16 with Activation Checkpointing