KOLM-Alpha — Kuramoto Oscillator Language Model

Delon Swartz — AI Researcher | Engineer

KOLM-Alpha is, to our knowledge, the first language model whose per-token processing layers are networks of coupled Kuramoto oscillators, trained natively from scratch. Each layer uses standard causal attention for routing between tokens and a Kuramoto block for processing: H oscillators, each a unit vector on a sphere, relax over K settling steps under trained pairwise couplings, conditioned on the attention output. The oscillator block replaces the Transformer's feed-forward network; attention is retained.

The result

Under a strictly controlled twin protocol — identical tokenizer, data, data order, context length, optimizer, schedule and seed, with only the feed-forward slot differing — KOLM-Alpha is compared against TMT (Transformer Model Twin), a standard attention + MLP model. Both were trained from scratch on 11.3M tokens of TinyStories on a single consumer laptop.

TMT leads early; KOLM-Alpha closes the gap steadily, crosses near 7M tokens, and finishes ahead — with 3.5% fewer parameters.

Model	Parameters	Val loss	Perplexity
KOLM-Alpha	16,916,224	2.6946	14.80
TMT (Transformer twin)	17,538,816	2.7188	15.16

Both models generate coherent TinyStories-grade prose. Prompt "Once upon a time there was a little red shoe":

KOLM-Alpha: …The little boy would come home and play with his friends. One day, the little girl was playing with the big castle in the sky…

Why it matters

A Transformer computes with a fixed number of operations per token. A Kuramoto block computes by iterative settling — the oscillator field relaxes toward a configuration over K steps — which opens a second axis of scale that costs no additional memory, only settling time. KOLM-Alpha establishes the first point: at matched parameters, oscillatory processing is not merely competitive with a Transformer feed-forward network — it edges it out.

Files

File	Purpose
native_kolm.py	model + twin trainer (--arch kolm / --arch transformer)
kuramoto_torch.py	the Kuramoto block (parity-tested to 2.2e-16 vs a NumPy reference)
kolm.py, olm.py	pure-NumPy reference core with hand-derived, gradient-checked backprop
sample_native.py	generation from the trained weights
native_kolm.pt	KOLM-Alpha weights (16.9M)
native_transformer.pt	TMT weights (17.5M)
tiny8k.json	tokenizer (required to run the model)
curve_kolm.csv, curve_transformer.csv	validation-loss curves

Run it

python3 -m venv --system-site-packages .venv
.venv/bin/pip install torch tokenizers

# generate from the released weights
.venv/bin/python sample_native.py --prompt "Once upon a time"

# verify the hand-derived gradients of the NumPy reference (no PyTorch needed)
python3 kolm.py --gradcheck

Reproduce the comparison

.venv/bin/pip install transformers accelerate
.venv/bin/python native_kolm.py --prep                 # build tokenizer + data
.venv/bin/python native_kolm.py --arch kolm        --steps 5500
.venv/bin/python native_kolm.py --arch transformer --steps 5500

Roadmap

KOLM-Alpha is the first release. In progress:

• KOLM-Beta — a larger model with frustrated coupling and a broader training corpus, grown from Alpha via function-preserving depth growth.
• KOLM-Chat — an instruction-tuned conversational KOLM, exposing the settling depth as a user-facing "think-harder" control.

Details

• Architecture: 8 layers, d=384, 6 heads, context 256, 8k byte-level BPE. Feed-forward slot = Kuramoto block (H=320 oscillators on S³, K=4 settling steps, 32 order-parameter readouts).
• Training: AdamW, lr 1e-4 cosine, gradient clip 1.0, single seed.
• Hardware: one Apple-silicon laptop (MPS), ~6 hours.
• Status: research preview. The headline comparison is single-seed; a second seed is queued.

License

Apache-2.0.

Acknowledgements

Builds on the Kuramoto-oscillator formulation of AKOrN (Miyato et al., ICLR 2025) and the TinyStories corpus (Eldan & Li, 2023).