🚀 ViuAI-UltraDeep (32 Layers)
An ultra-deep ~8.96M parameter MoE language model with DeepSeek-V4 MLA, Kimi Fine-Grained MoE, and mHC Hyper-Connections.
ViuAI-UltraDeep pushes the limits of sequential reasoning by stacking 32 Transformer Blocks with:
- 32 Reasoning Layers (Depth): Equivalent depth to frontier 8B models (Llama-3 has 32 layers).
- Multi-Head Latent Attention (MLA): Low-rank Query & KV Joint Compression (80%-90% KV-cache reduction).
- Fine-Grained MoE (DeepSeek / Kimi Style): 1 Always-Active Shared Expert + 8 Routed Experts (Top-2 active per token).
- Manifold-Constrained Hyper-Connections (mHC): Learnable residual scale vectors per layer that preserve gradient fidelity across all 32 layers.
- 52.6% MoE Compute Sparsity: Total capacity is
8.96M parameters, but compute cost per token is only **4.25M active parameters**! - Muon Optimizer: Newton-Schulz 5th-order orthogonalization for 1.5x-2x faster training convergence.
- Strict Weight Tying: Input embedding and output LM head share weights.
📊 Model Specifications
| Hyperparameter | Value | Description |
|---|---|---|
| Layers (Depth) | 32 | 32 Sequential Thinking Steps per Token |
| Total Model Parameters | 8,964,736 (~8.96M) | Full knowledge capacity across all 32 layers & experts |
| Active Parameters per Token | 4,246,144 (~4.25M) | 1 Shared + 2 Routed Experts executed per layer |
| Compute Sparsity | 52.6% | Over half the expert parameters bypassed per token |
| Memory Footprint (FP32) | ~34.20 MB | Extremely compact for a 32-layer LLM |
| Hidden Size ($d_{\text{model}}$) | 128 | Backbone representation dimension |
| Attention Mechanism | DeepSeek MLA | Multi-Head Latent Attention |
| Query LoRA Rank ($q_{\text{lora}}$) | 48 | Low-rank Query compression |
| KV LoRA Rank ($kv_{\text{lora}}$) | 32 | Low-rank Joint KV compression |
| Shared Experts | 1 per layer | Preserves core grammatical and semantic structures |
| Routed Experts | 8 per layer | Specialized fine-grained subnetworks |
| Top-K Routing | 2 per layer | Dynamic sparse gating |
| Expert FFN Dimension | 64 | SwiGLU mini-expert hidden dimension |
| Vocabulary | 4,096 | Custom BPE with Weight Tying |
| Context Length | 512 tokens | Supported sequence length |
| Residual Architecture | mHC | Learnable Hyper-Connection residual scaling |
| Training Optimizer | Muon + AdamW | Newton-Schulz matrix polar decomposition |
🚀 Quick Start
import torch
from viuai.model.config import ViuAIConfig
from viuai.model.viuai_model import ViuAIForCausalLM
# Load 32-layer model
config = ViuAIConfig(num_layers=32)
model = ViuAIForCausalLM(config)
print(f"Total Parameters: {model.count_parameters():,}")
# Output: 8,964,736 (~8.96M)
print(f"Active per Token: {model.count_active_parameters():,}")
# Output: 4,246,144 (~4.25M)
Built with ❤️ by ViuAI