geolip-rotary-experiments

Weights and result records from one day of experiments (2026-10-22) on rotary position encodings used as structural addresses. In these experiments a token's rotary phase comes from where it sits in a fixed structure (which compartment, which slot), not only from its index in the sequence. Small members, each trained alone, are merged into one frozen core behind a gate that reads that address.

Everything was trained from scratch on synthetic sequence tasks, mostly with a 4-layer, width-128 causal transformer (4 heads, context 256). Unless a row says otherwise, every result is two seeds, and the pass marks were written down before each run.

What is here:

folder	contents
weights/	475 PyTorch state dicts (1.7 GB): every core, member stack and head the experiments trained, failures and controls included
reports/	the result tables (*_tables.md), one JSON record per run (phase*/, speed/), and the run logs

The code that trained and measured these files is not hosted here; this repository holds weights and results only.

The idea in two formulas

For rotate-half RoPE with pair frequencies ${\omega }_j\backslash omega\_j$, two identical unit vectors at positions $mm$ and $nn$ score

$$K(m-n)=\frac{1}{P}\sum _{j=1}^P\cos ({\omega }_j(m-n))K(m-n)\; =\; \backslash frac\{1\}\{P\}\backslash sum\_\{j=1\}^\{P\}\backslash cos\backslash big(\backslash omega\_j\backslash ,(m-n)\backslash big)$$

This is the address kernel. Its main lobe is the resolution, its side lobes are what a wrong address leaks, and the number of positions it can hold apart is its capacity. A query that asks for an address is rotated by that address, and a key by its own.

Members are merged behind an amplitude gate that reads the requested address $rr$ against each member's own address $ii$, level by level:

$$u_i=U\prod _l{k}_l(r_l,i_l),w_i=\frac{\sinh u_i}{1+\sum _j(\cosh u_j-1)}u\_i\; =\; U\backslash prod\_\{l\}\; k\_l(r\_l,\; i\_l),\; \backslash qquad\; w\_i\; =\; \backslash frac\{\backslash sinh\; u\_i\}\{1\; +\; \backslash sum\_j\; \backslash left(\backslash cosh\; u\_j\; -\; 1\backslash right)\}$$

A member whose address does not match gets $u_i=0u\_i\; =\; 0$, and $\sinh 0=0\backslash sinh\; 0\; =\; 0$: it abstains outright, and no member is chosen over another. A matching member gets $w_i\approx \tanh Uw\_i\; \backslash approx\; \backslash tanh\; U$.

The experiments, in order

part	question	size	result (two seeds)
Rotary catalogue (phase A)	which rotary forms work on relative, absolute and compartment lookups	24 forms, 144 runs	NTK-scaled RoPE has the best reach at RoPE's cost; the Cantor rotaries, SinusoidalRoPE at its defaults and an additive quad-rotary embedding fail
Fixed-position prototypes (phase B)	does a query rotated to the address it asks for find it?	62 runs	yes, with every code: compartment lookups 1.000, and absolute slots .97-1.000 at four times the trained length (dynamic forms ~.28 there)
Merging members (phases C, C2a, C2b)	do members trained alone merge on a frozen core without loss or leak?	92 cells	with the structural request: merged within .008 of each member alone, nothing leaked (KL at most .005); ungated members wreck each other (merged .05-.26)
Mixed sequences (phase D)	relative and fixed-position questions in one sequence	24 runs	both kinds answered 1.000
Band capacity (phase E)	how many members one band of anchors holds	16 compartments, 14 members	exact up to its count of orthogonal anchors; a squeezed band leaks in proportion to the anchors' overlap (KL 0 / .6-.8 / 1.6-2.6 at overlap 0 / .56 / .85)
Follow-ups (phases F, F4)	shuffled content, members attached mid-stack, the full normalizer	38 runs	the anchor band carries a compartment's identity when content cannot (1.000 vs .17); mid-stack members work (1.000 vs .04-.07)
Speeds	what every rotary costs at 4k-64k tokens	three timed runs	the house rotary is 3.7-5.3% of a block at 4k and 1.5% at 64k; the anchored rotary costs 1.24-1.27x the house rotary at 4k and about the same once compiled
Screen S1	does address capacity multiply?	2 seeds	one gate over the product of two 9-anchor level kernels addresses 64 compartments exactly in 18 dimensions, where one flat band needs 65
Screen S2	three levels, and how many members one gate holds	2 seeds	512 addresses exact in 27 dimensions; a normalizer that charges resting members nothing removes the member ceiling
Screen S3	where the request comes from when a query names only a key	2 seeds	an index the core writes into the sequence is exact (0 wrong of 65,536 entries) and gives the exact gate; requests read off attention fail
S3 follow-ups	the pointing block alone, a learned address head, why the key lookup trains slowly	2 seeds	below
Screen S3L	a learned address request that does not leak	2 seeds, 2 head forms	a core trained together with its address head gives an exact request (nothing leaked, every mark kept); a floor on a frozen-core head cannot; members on the new cores need a longer member schedule (below)

The S3 follow-ups in more detail

One block of attention is a working request when the core points at the right record. One seed's core points its second block at the asked record (three of four heads at .955). Read from that block alone, the request gives the right member .41 of a full match (.73 in the flat form). It keeps the members' answers (.97) with nothing leaked (.003 of an owned write). The same request averaged over all four blocks gave .010 and silenced every member. The other seed's core points at the next record, so the same read hands the neighbour the weight.

A head trained on a frozen core reads the address as a whole, not digit by digit, and leaks. After the pointing block, a linear head with one sigmoid per compartment names the asked compartment 97% of the time (blocks 2-4). A per-digit head reads only the coarse digit (.96-.98) and not the finer two (.23-.32). The flat head's scores are each judged alone: the largest wrong score is only .03-.04, but 255 such writes add up on a question no member owns (those answers fall to .39-.72 at 256 members). Attention does not leak this way, because its softmax splits one shared budget.

Why the key-to-value lookup trains slowly (the answer job alone, 12,000 steps):

layout	answers right, seed 1 / 2	training loss at 10,000 steps
the S3 layout	.085 / .084	3.17 / 3.18 (= log of the row's ~25 distinct values: it knows the row's values, not which one a key names)
no index sector, the same number of answered questions	.084 / .087	3.18 / 3.16
no index sector, 1.84x the answered questions	.558 / .268	1.50 / 2.45
the questions asked like an index (in key order, each answer shown after its question), values repeating	.173 / .174	2.44 / 2.43
the same, each value unique in its row and its own token	.9999 / 1.000	0.00 / 0.00 (formed between steps 2,000 and 8,000)

Removing the index sector changes nothing at matched supervision; more answered questions per step breaks the plateau slowly; asking the questions like an index entry with unique answers forms the lookup in a few thousand steps, which is how the core's own index formed.

Screen S3L: a learned address request that does not leak

The S3 follow-ups left one problem. A head trained on a frozen core names the asked compartment 97% of the time, but its request leaks in two ways: each of the 511 wrong compartments keeps a small score of its own, and on a question no member owns those small writes add up (the sum); and on about 3% of questions the head's largest score is a wrong compartment, which hands that member most of a full write (a wrong request). The fix had to keep the gate's rule: no member is compared with another.

An absolute floor on the frozen-core head. Each member's factor comes from its own score and one fixed number:

$$f(c)=\frac{\max (0,s(c)-\tau )}{1-\tau }f(c)\; =\; \backslash frac\{\backslash max\backslash big(0,\backslash \; s(c)\; -\; \backslash tau\backslash big)\}\{1\; -\; \backslash tau\}$$

with $\tau \backslash tau$ the 90th, 99th or 99.9th percentile of the per-question largest wrong score, measured on calibration rows the read never sees. A score at or under the floor gives exactly 0, and $\sinh 0=0\backslash sinh\; 0\; =\; 0$. The floor removes the sum (the leaked write falls from .51 to .03 / .005 / .0004 of an owned write) but not the wrong requests: a floor low enough to keep the owners still passes the confident wrong scores, and one high enough to stop them silences up to 43% of the owners. No floor height keeps every mark on a frozen core.

A core trained together with its address head. The core is trained by the same recipe plus one term, the head's binary cross-entropy at the questions, summed over its outputs:

$$L=\frac{1}{\mathrm{\mid }S\mathrm{\mid }}\sum _{t\in S}\mathrm{C}\mathrm{E}(y_t,p_t)+\lambda \frac{1}{Q}\sum _q\sum _o\mathrm{B}\mathrm{C}\mathrm{E}(z_{q,o},{\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t}}_{q,o}),\lambda =1L\; =\; \backslash frac\{1\}\{|S|\}\backslash sum\_\{t\; \backslash in\; S\}\backslash mathrm\{CE\}(y\_t,\; p\_t)\; \backslash ;+\backslash ;\; \backslash lambda\backslash ,\backslash frac\{1\}\{Q\}\backslash sum\_\{q\}\backslash sum\_\{o\}\backslash mathrm\{BCE\}\backslash big(z\_\{q,o\},\backslash \; \backslash mathrm\{target\}\_\{q,o\}\backslash big),\; \backslash qquad\; \backslash lambda\; =\; 1$$

in two forms: one sigmoid per compartment (512 outputs) and one sigmoid per digit (3 x 8 outputs). On all four cores (two forms, two seeds) the head names the asked compartment on every one of the read's 158,772 questions, its wrong scores stay near ${10}^{-7}10^\{-7\}$ or below, and its request keeps all three marks in every cell with a leaked write of about ${10}^{-8}10^\{-8\}$ of an owned write, with or without the floor. The term also makes the key-to-value lookup form in 4,000-8,000 steps (the cores without it needed 48,000-96,000).

core	the head right (158,772 questions)	leaked write / owned write, largest	cells keeping all three marks (plain head; floored at three heights)	members alone after 4,000 / 16,000 member steps
one sigmoid per compartment, seed 1	1.00000	2e-8	9 of 9; 9, 9, 9	.902 / 1.000
one sigmoid per compartment, seed 2	1.00000	7e-9	9 of 9; 9, 9, 9	.988 / 1.000
one sigmoid per digit, seed 1	1.00000	7e-8	9 of 9; 9, 9, 9	.637 / .9998
one sigmoid per digit, seed 2	1.00000	5e-8	9 of 9; 9, 9, 9	.157 / 1.000

The cost is in the members, and it is their schedule. Members trained on these cores by the night's member recipe (4,000 steps, the learning rate decaying to a tenth) learn more slowly: every member run starts on a plateau, a uniform guess over the 64 values, and on the new cores the members are still on it when the learning rate has fallen, so alone they read .16-.99 (on the plain cores .998-1.000). The same recipe over 16,000 steps, with the same data and starting weights, has every member at 1.000 alone and the merged answers at 1.000 with every mark kept. Why the plateau lasts longer on these cores is open: the answer's confidence, the state's size and how much of a member's input carries the value were measured and do not explain it.

Caveats

• Small synthetic beds: a 4-layer width-128 core, full attention at context 256, 64 value tokens, at most 512 compartments.
• Two seeds per cell; several results differ by seed in how the core solved the task (which record its attention points at).
• The member weights only make sense with their frozen cores (both are here; the file names pair them).

File names

<phase>_<form>[_<position mode>]_<task>_s<seed>_st<steps>_d128L4.pt for cores (phases A, B, D, F; screens S1-S3). Member stacks name the gate, the site and the core they belong to (C_* and E_* end in _members.pt; S1_members_*, S2_members_*, S3_members_*). smoke_* are the bench's 300-step smoke tests. S3G_* are the S3 follow-ups: the address heads (S3G_head_*, S3G_HEADP*, S3G_HEADF*) and the layout runs (S3G_rope_p_struct_<layout>_*). S3L_* are screen S3L: the cores trained with their address heads (S3L_JF_* one sigmoid per compartment, S3L_JP_* one sigmoid per digit; each head in the matching *_jhead.pt), their member stacks (S3L_members_m256p_<core>_st4000.pt and _st16000.pt, the member steps), and the trainer parity runs (S3L_R3_*). Every run's JSON record in reports/ names its weights file, its configuration and its measured results.

Load a state dict with torch.load(path, map_location="cpu", weights_only=True).