ARO-BabyLM-1B-Cooldown: Asymmetric Relational Operator

The Ultimate Present

日本語概要

A 116.6M-parameter causal language model trained from scratch for exactly one billion words of exposure over the official BabyLM 2026 Strict corpus.

ARO stands for Asymmetric Relational Operator. Asymmetric means that forward and reverse relations remain distinct. Relational refers to learned latent modes that compare the evolving Present state with several internal evidence sources. Operator means that these relations actively condition the feed-forward transformation and reorient the next Present state.

This repository contains the performance-directed cooldown continuation of the verified ARO 500M checkpoint. It preserves the model, tokenizer, corpus order, seed, AdamW state, and optimizer trajectory through 500M words, then uses a decaying learning rate and a conservative context curriculum through exactly 1B words.

Research status — 2 September 2026: pretraining and the official BabyLM 2026 Full zero-shot and Reading evaluations are complete. This cooldown branch is released as a training-recipe study. The Original 1B model is the recommended ARO checkpoint for general evaluation and use.

Model and training facts

Item	Value
Parameters	116,647,315
Architecture	causal ARO language model
Vocabulary	16,384 byte-level BPE tokens
Physical layers	11
Hidden / FFN size	768 / 2,752
Maximum supported context	1,024
Word exposure	exactly 1,000,000,000
Unique corpus	exactly 100,000,000 words
Subword prediction tokens processed	1,513,881,600
Optimizer steps	11,550
Seed	1337
Hardware	one NVIDIA RTX 3090

The model was trained only on the official detoxified English BabyLM 2026 Strict corpus. The fixed 100M-word corpus was traversed ten times. The tokenizer was trained only on that training corpus.

Cooldown continuation

The 500M parent checkpoint was imported with its complete AdamW state. From 500M to 800M words, training retained sequence length 256 while the learning rate followed a half-cosine decay from 8.056641995e-4 to 1e-4. From 800M to 1B, sequence length was 512 and the learning rate decayed to approximately 1e-5. Global prediction tokens per optimizer step remained 131,072.

This is a performance-directed recipe bundle selected from the observed 500M learning curve. It is not a single-factor causal experiment. The repository preserves the contracts and lineage needed to distinguish this continuation from the earlier high-learning-rate 1B run.

The Original and Cooldown runs share the same model, tokenizer, corpus order, seed, parent weights, and AdamW state through 500M words. Their continuation recipes then diverge. The Original branch moves from context length 256 to 512 at 600M words and to 1,024 at 800M, while retaining a learning-rate floor of 6e-4. The Cooldown branch stays at context length 256 through 800M, moves to 512 for the final 200M words, and decays the learning rate to approximately 1e-5.

Full benchmark results

Both checkpoints were evaluated with the same official BabyLM 2026 Full zero-shot pipeline. The Cooldown result manifest contains 20 files, all of which passed SHA-256 readback.

Benchmark	Original 1B	Cooldown 1B	Cooldown minus Original
BLiMP	75.8894	74.2984	-1.5910
BLiMP Supplement	65.3623	63.5337	-1.8286
EWoK	55.2991	55.2789	-0.0202
Entity Tracking	19.9495	18.4643	-1.4852
COMPS	57.2323	56.3418	-0.8905
GlobalPIQA	38.6650	38.0922	-0.5728
Simple mean of the six metrics	52.0663	51.0016	-1.0647
Reading	3.0449	5.3481	+2.3033

When inserted into the 55 public Strict rows displayed by the BabyLM 2026 leaderboard on 2 September 2026, the six-metric mean places Original at reference position 17/56 and Cooldown at 23/56. These are comparison positions for the six completed zero-shot metrics. SuperGLUE and AoA are incomplete, so neither checkpoint has an official BabyLM overall rank.

The cooldown branch achieved a lower observed training cross-entropy under its shorter-context curriculum, while all six zero-shot metrics declined. Reading improved substantially. The result shows a real tradeoff: aggressive learning-rate decay and delayed context growth produced a model that fit its training stream more closely but transferred less well across the six broad zero-shot evaluations measured here.

Learning rate and context length changed together after 600M words. The comparison therefore measures the complete continuation recipe rather than assigning the outcome to either change alone. The machine-readable values and evaluation identities are available in audit/evaluation/COOLDOWN_VS_ORIGINAL_6METRIC.json.

Architecture summary

ARO maintains one evolving full-width Present state. Eleven blocks combine:

• causal local/global grouped-query attention;
• recurrent GatedDeltaNet memory;
• routed reports from earlier depth;
• relation-conditioned SwiGLU using twelve learned relation modes;
• Deep Present rereading of fixed past-side evidence from an updated Present;
• a bounded state write and an adaptive feature path;
• training-only multi-horizon Future Cone predictors.

Inference is strictly causal and uses only the visible prefix. Future Cone heads shape the representation during training but do not expose future tokens at inference.

ARO's conceptual origin combines an account of time as a panorama centered on the observer's present with the Buddhist principle of dependent origination. These sources motivated an executable question: can one evolving Present become more precise by relating several distinct forms of past evidence rather than merely adding them together?

Technical report

The architecture and its design rationale are documented in:

Yusuke Maeda. Evidence-Conditioned Present-State Formation: An Architecture for Compact Language Models. Technical note, Zenodo, 30 August 2026. https://doi.org/10.5281/zenodo.22167627

@techreport{maeda2026evidence,
  author      = {Yusuke Maeda},
  title       = {Evidence-Conditioned Present-State Formation: An Architecture for Compact Language Models},
  year        = {2026},
  month       = aug,
  institution = {Zenodo},
  doi         = {10.5281/zenodo.22167627},
  url         = {https://doi.org/10.5281/zenodo.22167627}
}

Integrity and audit material

The audit/ directory contains the run contract, authorization record, schedule evidence, source freeze, corpus manifest, parent lineage, terminal run status, native checkpoint integrity records, and canonical training log. It does not contain the training corpus or optimizer-bearing native checkpoints.

Final native checkpoint SHA-256:

2351528cae2c57eb186722acdf990f4c1a87420fe8e8c8ff3f837cb202b43609

Exported model.safetensors SHA-256:

f1b282002dfb93e5be77fb9bf5ec8cbf09701b7868d139655979b8d8d450f60b

Tokenizer SHA-256:

8edfa792838717e0940c309880a71acf1b276d34981576212b0fffaf3cf1d909

Usage

This model contains custom Transformers code. Review the repository source before enabling it.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "AwakeningOS/ARO-BabyLM-1B-Cooldown"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

The model is a base causal language model, not an instruction-tuned assistant.

Evaluation scope

The reported evaluation uses official BabyLM 2026 code commit 6f825c291e2c4c78ad33b1935fd64d45f52642dc and data revision 8d52da9424a9ff30b9e8266c4f751aba9c504233. Both ARO checkpoints are single-seed runs. The public leaderboard entries differ in architecture and training implementation, so the reference positions describe observed scores rather than an isolated architecture comparison.

This work reproduces the BabyLM 2026 Strict conditions after the competition deadline. It is an independent research release, not an official competition entry or award claim.