TalosDDM

254,272 parameters 64 hidden size 2 layers 4 attention heads 2 KV heads 1,024 vocabulary 512 max sequence length FP32 PyTorch 2.11.0+cu128 CUDA 12.8.1 NVIDIA Tesla T4

Training Loss

Step 0 — 6.9177 Step 25 — 5.3568 Step 50 — 3.9134 Step 75 — 2.6434 Step 100 — 1.6195 Step 125 — 1.0475 Step 150 — 0.5186 Step 175 — 0.2652

MoE Configuration

128 hidden size 3 layers 8 attention heads 4 KV heads 16 head dimension 2,048 vocabulary 16 experts 3 active experts 1 shared expert