Fibration/babylm-force-8h-12L

Can Higher Order Attention Help Small Language Models?

BabyLM 2026 submission · Joseph Chan

We investigate whether higher-order interactions in attention can improve small language models trained under severe data constraints. We introduce and evaluate a force layer inside a Transformer, comparing against standard Transformer baselines on the BabyLM 10M word setting.

A force layer is an attention layer where the attention score has a quadratic correction term to the first order dot product attention score prior to applying softmax. This model has 6 prefix attention layers, one force layer, followed by 5 suffix attention layers.

The model hyperparameters are:

• 8 heads
• 64 head dimension
• 512 hidden dimension
• 4 * 64 * 16 = 4096 per head second order parameters
• 12 layers
• seed 3407
• 46.21M total parameters
• BPE 16k tokenizer
• 16,384 vocab
• 10 epochs of training on 10M BabyLM Strict-Small dataset (stopped early with 5% of last epoch buffer)
• 162430976 tokens seen
• 99538153 estimated words seen