captionbert-8192-rotary
A controlled positional screen of CaptionBERT v2: the learned-position replication (A0), plain token RoPE (A1), sentence-address rotary (A2), and its anchor ablation (A3). Every arm is trained from scratch against the same precomputed five-teacher consensus. Inference uses the tokenizer and one student; teacher models are not loaded.
This is a short screen, separate from the original v2 full training run. Both seeds and all arms, including weaker results, are retained. No single winner is selected automatically.
Loading
Install the tagged package in an isolated Python 3.11–3.13 environment:
pip install git+https://github.com/AbstractEyes/captionbert-trainer.git@v0.2.0
from captionbert_rotary.inference import SentenceEncoder
encoder = SentenceEncoder.from_pretrained("AbstractPhil/captionbert-8192-rotary",
run="experiments/817daa9e66a8119dc6b18a9b63fad32490d1c891e5f26750e91125d9e5e14e6c/A2-seed1",
revision="4e75920dfe6ddd820db321608a853c2097533f4e")
embeddings = encoder.encode(["A cat watches the street.", "A person walks nearby."])
# embeddings: (2, 768), L2-normalized
Choose an explicit arm and seed. A2-seed1 above is a loading example, not a claim that it is best. Model code, requirements, tests and notebooks live in captionbert-trainer.
The technical companion describes the measurement controls, report fields and limits.
Benchmark
Spearman correlation on the same eight test tasks as the v2 harness; mean is the unweighted task average. The benchmark uses its original 64-token ceiling.
| arm / seed | STS-B | SICK-R | STS12 | STS13 | STS14 | STS15 | STS16 | BIOSSES | mean |
|---|---|---|---|---|---|---|---|---|---|
| A0 / 1 | 0.4531 | 0.6199 | 0.4476 | 0.4354 | 0.4537 | 0.6350 | 0.4667 | 0.5083 | 0.5025 |
| A0 / 2 | 0.4557 | 0.6224 | 0.4292 | 0.4443 | 0.4470 | 0.6445 | 0.4750 | 0.4291 | 0.4934 |
| A1 / 1 | 0.5057 | 0.6330 | 0.4756 | 0.4316 | 0.4764 | 0.6601 | 0.5166 | 0.4693 | 0.5211 |
| A1 / 2 | 0.4965 | 0.6314 | 0.4769 | 0.4673 | 0.4736 | 0.6689 | 0.5148 | 0.3934 | 0.5153 |
| A2 / 1 | 0.5248 | 0.6339 | 0.4817 | 0.4665 | 0.4946 | 0.6665 | 0.5206 | 0.4607 | 0.5312 |
| A2 / 2 | 0.4864 | 0.6328 | 0.4706 | 0.4405 | 0.4649 | 0.6756 | 0.4949 | 0.4872 | 0.5191 |
| A3 / 1 | 0.5252 | 0.6383 | 0.4741 | 0.4755 | 0.4931 | 0.6685 | 0.5251 | 0.4983 | 0.5373 |
| A3 / 2 | 0.4845 | 0.6352 | 0.4528 | 0.4538 | 0.4751 | 0.6762 | 0.5010 | 0.4168 | 0.5119 |
The published full-run v2 mean was 0.6077 after 26.9M rows and 52,548 steps. That is historical context, not the matched control for this screen; the matched control is A0 here.
Geometry
Effective rank is the participation ratio of centered variance. Self-cosine is the mean off-diagonal cosine, an isotropy readout. Cosine to consensus measures mimicry, separate from task capability.
The following caption-domain measurements use the fixed held-out chunk-61 sample, separately from the benchmark sentences. The >128-token slice is sampled separately from that chunk.
| arm / seed | parameters | steps | held-out cosine | effective rank | self-cosine | seconds/step |
|---|---|---|---|---|---|---|
| A0 / 1 | 58,308,864 | 1952 | 0.7226 | 68.19 | 0.0191 | 0.400 |
| A0 / 2 | 58,308,864 | 1952 | 0.7478 | 71.79 | 0.0181 | 0.400 |
| A1 / 1 | 54,114,560 | 1952 | 0.7459 | 71.34 | 0.0194 | 0.464 |
| A1 / 2 | 54,114,560 | 1952 | 0.7615 | 74.13 | 0.0171 | 0.464 |
| A2 / 1 | 54,114,560 | 1952 | 0.7427 | 71.87 | 0.0204 | 0.464 |
| A2 / 2 | 54,114,560 | 1952 | 0.7635 | 73.77 | 0.0158 | 0.465 |
| A3 / 1 | 54,114,560 | 1952 | 0.7429 | 69.80 | 0.0205 | 0.465 |
| A3 / 2 | 54,114,560 | 1952 | 0.7622 | 73.08 | 0.0169 | 0.465 |
| arm / seed | >128-token cosine | effective rank | self-cosine | measured rows |
|---|---|---|---|---|
| A0 / 1 | 0.6549 | 53.25 | 0.0312 | 5000 |
| A0 / 2 | 0.6831 | 50.99 | 0.0610 | 5000 |
| A1 / 1 | 0.6730 | 54.01 | 0.0358 | 5000 |
| A1 / 2 | 0.6862 | 53.85 | 0.0672 | 5000 |
| A2 / 1 | 0.6723 | 55.57 | 0.0345 | 5000 |
| A2 / 2 | 0.6879 | 53.59 | 0.0551 | 5000 |
| A3 / 1 | 0.6740 | 55.14 | 0.0344 | 5000 |
| A3 / 2 | 0.6834 | 52.04 | 0.0611 | 5000 |
Rotary measurements
The fixed frozen-model protocol measures address kernels, simplex coherence and its explicit constant lift, periodic aliases versus the sentence clamp, identical-content oracle address lookup, controlled address changes, q/k-band removals, and per-layer/head energy and attention-mass changes.
The encoder rotates tokens to their own coordinates. It has no learned request head or AMOE member gate. Code-only oracle lookup and embedding sensitivity are not learned semantic-retrieval accuracy or task improvement. The raw anchor code is a regular simplex; the encoder does not fix the constant coordinate of the orthogonal lift.
| arm / seed | repeated baseline max error | sentence-ID collapse, mean L2 change | anchor q/k removal, mean L2 change |
|---|---|---|---|
| A0 / 1 | 0.00000000 | n/a | n/a |
| A0 / 2 | 0.00000000 | n/a | n/a |
| A1 / 1 | 0.00000000 | 0.000000 | 0.000000 |
| A1 / 2 | 0.00000000 | 0.000000 | 0.000000 |
| A2 / 1 | 0.00000000 | 0.272284 | 0.116291 |
| A2 / 2 | 0.00000000 | 0.318407 | 0.135013 |
| A3 / 1 | 0.00000000 | 0.000000 | 0.000000 |
| A3 / 2 | 0.00000000 | 0.000000 | 0.000000 |
A0 has no rotary bands (n/a). A1/A3 have no anchor band, so anchor-removal effects are structural negative controls. Full probe inputs, checkpoint/source hashes, dtype/device, all interventions and every layer/head are in each run's rotary_diagnostics.json or additive analyses/captionbert-rotary-mechanisms-v1/ report. The frozen repeat measures diagnostic-call precision, not training hardware-repeat uncertainty.
How it was built
| part | screen recipe |
|---|---|
| arms / seeds | A0, A1, A2, A3 / 1, 2 |
| training chunks | [0, 1, 2, 3] |
| held-out chunk | 61 |
| epochs / contrastive batch | 2 / 2048 |
| optimizer | pure Adam, weight decay zero; peak LR 0.0006 |
| schedule | 2000-step linear warmup from 1%, then cosine |
| training token ceiling | 256 |
| objective / pooling | symmetric InfoNCE (T=0.07) + MSE; masked mean; L2-normalized 768-d output |
Captions and consensus tensors are paired by flat row index, with exact row-count checks. Sentence indices advance after ., ! or ?; the next token resets its sentence-local slot. A2 has content 8, slot 12 (wavelengths 3–256) and anchor 12 pairs per 64-d head; A3 has content 8 and slot 24. A1 uses all 32 pairs on global token index.
Known limits
- The short screen spends most of its approximately 2,000 steps in the original 2,000-step warmup. It does not reproduce v2's full-run scale.
- Training is capped at 256 tokens. The model name and A0's 8,192-position table do not establish 8k-context quality.
- Sentence IDs clamp at 24. Unclamped rotary codes repeat every 25 addresses; neither is unlimited sentence capacity.
- Caption-domain consensus agreement and STS capability are separate measurements. These five teachers also constrain the target's shared geometry.
- Paired seeds expose seed dependence; a settled improvement additionally needs the applicable uncertainty/repeat evidence. A large band-removal response alone does not establish useful structure.
Output convention
The package's SentenceEncoder.encode returns a (batch, 768) normalized tensor. This repository contains multiple explicit run artifacts; it is not a root-level Transformers AutoModel checkpoint.
Provenance
Experiment 817daa9e66a8119dc6b18a9b63fad32490d1c891e5f26750e91125d9e5e14e6c; result files read at Hub revision 4e75920dfe6ddd820db321608a853c2097533f4e. Model-card organization follows the v2 card. Original results, every arm/seed, weights and analysis JSON remain under their immutable experiment paths.
Citation
@misc{captionbert_rotary_2026,
title = {CaptionBERT Sentence-Address Rotary Screen},
author = {AbstractPhil},
year = {2026},
url = {https://huggingface.co/AbstractPhil/captionbert-8192-rotary}
}