adpretko/celerity-271m-8k-tema-ablation-ad0p2-tema4

Celerity 271M 8K — TEMA ablation — ad0.2_tema_mult4

Official Celerity 271M / 8K tau-EMA-ablation checkpoint converted from Cerebras CS format to Hugging Face format.

Training provenance

• Original experiment: ad0.2_tema_mult4
• Source checkpoint: checkpoint_13773.mdl
• Attention dropout rate: 0.2
• Attention-dropout schedule: constant
• Peak learning rate: 0.15
• Tau-EMA multiplier relative to 0.1745: 4
• Weight decay: 0.0006934653580420589
• tau_ema: 0.698
• Global train batch size: 48
• Validation batch size: 32
• Training steps: 13773
• Maximum sequence length: 8192
• Position embedding type: ALiBi
• Residual dropout rate: 0.0
• Stochastic depth: 0.0
• LayerDrop: 0.0
• Source runtime experiments: cbcore 2.6.0
• Converter commit: 0e3d5d375695293479df9d2a3717f05f71a345b4

This tau-EMA ablation holds learning rate at 0.15 and global train batch size at 48 while varying tau_ema relative to the 0.1745 baseline. Weight decay is adjusted accordingly.

Attention dropout, tau_ema, and weight decay are training-time hyperparameters. Their effects are represented in the learned checkpoint weights. Hugging Face evaluation is performed with dropout disabled by model.eval().

The model uses custom Celerity Hugging Face modeling code and should be loaded with trust_remote_code=True.