4995 paper — Block AttnRes vs PreNorm baseline (1B)

Replication of Attention Residuals (Kimi Team, arXiv:2603.15031) at 1B scale.

Models

•        PreNorm baseline, 1B params
  

•  Block AttnRes (S=8, 9 blocks), 1B params
  

Architecture: 36 layers × 1536 hidden × 24 heads × 6144 FFN, ctx=1024, GPT-2 BPE vocab (50304).

Trained for 15000 steps × 98K tokens/step = 1.47B tokens of FineWeb-Edu sample-10BT, BF16, AdamW, cosine LR schedule, 8× B200 NVLink-5.

Each is a torch.save dict containing model+optimizer state and the training config (, ). Load with:

Full results, plots, and code in the project's git tree (not uploaded here).