ARO-BabyLM-1B: Asymmetric Relational Operator

The Ultimate Present

日本語概要

A 116.6M-parameter causal language model trained from scratch under the BabyLM 2026 Strict word-exposure budget.

ARO stands for Asymmetric Relational Operator. The name describes the architecture's central computation. Asymmetric means that forward and reverse relations are kept distinct rather than collapsed into one symmetric relation. Relational refers to the learned latent modes through which the model compares its evolving Present state with several internal evidence sources. Operator emphasizes that these relations do more than describe the state: they condition the feed-forward transformation and help reorient the next Present state.

The architecture maintains one evolving Present state where three kinds of evidence meet: exact causal retrieval from the visible context, a compressed recurrent memory of the past, and training-time pressure to preserve information useful for predicting several future horizons. Inference remains strictly causal and uses only the visible prefix.

ARO's conceptual origin lies in two sources: a near-death-experience account that described time as a single panorama centered on the observer's present, and the Buddhist principle of dependent origination, in which phenomena arise through chains of relation. These ideas were translated into testable computation: one evolving Present state, multiple forms of evidence arriving from the past, a second reading of the same past from an updated Present, twelve learned relation modes with forward and reverse structure, and training pressure directed toward several future horizons.

This repository is the BabyLM specialization of ARO. It contains the 1B-word-exposure checkpoint, a native 16,384-token byte-level BPE tokenizer, custom Transformers integration, and the architecture source required to reproduce inference.

Evaluation status — 1 September 2026: pretraining is complete and the final checkpoint has passed its SHA-256 integrity check. Official BabyLM 2026 full zero-shot and Reading evaluation is running. The table below reports the verified 500M intermediate checkpoint on the official fast checkpoint suite. Final-model results will replace the pending section after verification.

Highlights

• 116,647,315 parameters with tied input and output embeddings.
• 1,000,000,000 words of total training exposure, drawn from a fixed 100M-word corpus over ten passes.
• 1,513,881,600 subword tokens processed in 11,550 optimizer steps.
• A hybrid causal architecture combining local/global attention, GatedDeltaNet memory, routed depth reports, relation-conditioned SwiGLU, Deep Present recurrence, and a training-only multi-horizon Future Cone objective.
• A native 16,384-token byte-level BPE tokenizer, trained only on the official BabyLM 2026 Strict training split.
• Trained from scratch on a single NVIDIA RTX 3090 with BF16 autocast.

Verified intermediate results

BabyLM 2026 fast checkpoint evaluation at 500M words

Task	ARO 500M
BLiMP	78.45
BLiMP Supplement	64.80
EWoK	52.91
Entity Tracking	21.24
GlobalPIQA, parallel	23.30
GlobalPIQA, nonparallel	51.00
GlobalPIQA, mean of the two subsets	37.15
Reading, eye tracking	7.15
Reading, self-paced	2.86

The result bundle contains 18 files and passed a complete size and SHA-256 readback against its result manifest. This is a single-seed intermediate-checkpoint result and should be treated as a developmental measurement.

Context against published systems

The following table places the intermediate result beside published reference numbers. Evaluation suites and scopes changed across years; the table provides descriptive historical context.

Model	BLiMP	BLiMP Supplement	EWoK	Entity Tracking
ARO, 500M exposure, 2026 fast suite	78.45	64.80	52.91	21.24
GPT-2 Strict baseline, 2026 full suite	74.73	65.00	54.37	16.91
BLaLM Strict, 2025	74.49	60.40	53.91	21.99
Transformer Strict, 2025	72.44	62.00	53.36	20.63
BabyLlama Strict, 2024	73.10	60.60	52.10	—
LTG-BERT Strict, 2024	69.20	66.50	51.90	—

ARO's clearest intermediate strength is grammatical minimal-pair accuracy. Leaderboard rank, downstream transfer, and component-level causal attribution will be assessed from the final evaluation and later controlled studies.

Final 1B evaluation

Evaluation family	Status
BabyLM 2026 full zero-shot	Pending verified completion
Reading-time prediction	Pending verified completion
(Super)GLUE fine-tuning	Not yet evaluated
Age of Acquisition	Not yet evaluated

Architecture

ARO uses one full-width present state, h ∈ R^(B×T×768), throughout eleven physical layers. Each layer combines several complementary paths that specialize in exact retrieval, compressed memory, depth evidence, and relation-conditioned transformation.

token ids
   │
   ▼
16,384-way tied embedding × √768
   │
   ▼
11 × ARO block
   ├── causal local/global grouped-query attention
   ├── recurrent GatedDeltaNet memory
   ├── routed reports from earlier depth
   ├── relation-conditioned SwiGLU
   ├── ADS feature path
   └── bounded Present-state write
   │
   ▼
RMSNorm → tied vocabulary projection → next-token distribution

Causal local/global attention

Most layers use a 256-token causal window. Layers 3, 7, and 10 use global causal attention. The attention path has eight query heads, two key/value heads, and a head dimension of 64. RoPE supplies positional information.

Each layer also learns a query-dependent post-attention head gate and a shared residual 64×64 Value basis. The gate modulates the retrieved moments, while the Value basis provides a small learned coordinate transform shared across heads in that layer.

GatedDeltaNet memory

Attention is paired with a recurrent GatedDeltaNet path. It compresses the entire visible prefix into a continuously updated state, giving the model an all-past memory path alongside exact attention retrieval. The BabyLM configuration uses six memory heads of width 48, a 384-dimensional bottleneck, expansion factor 2, and a short convolution of width 4.

Deep Present

The attention calculation is refined for two internal loops. The transformed Value representation is held fixed while the current query and its headwise gate are renewed. This lets the model reconsider how the present state reads the same available evidence without introducing a second residual stream.

Routed depth reports

Starting at configured depths, a four-head router retrieves compact reports derived from earlier layer changes. These reports give later blocks selective access to depth history.

Relation-conditioned SwiGLU

The feed-forward path has width 2,752. Before the SwiGLU output is written back, the block constructs twelve relation modes from separately normalized evidence sources: the current state, attention output, recurrent-memory output, routed depth reports, displacement from the token embedding, and attention dispersion where available.

For present observation p and evidence source e_i, shared projections produce directional interactions:

F_i = L(p) ⊙ R(e_i)
R_i = R(p) ⊙ L(e_i)

Evidence weights are normalized across sources. Symmetric, antisymmetric, product, squared-difference, self, cross-source, and relative-magnitude features are combined into a twelve-dimensional relation state. A learned 12 → 4×2752 basis then modulates every SwiGLU gate and value unit independently. A bounded steering gain controls how the proposed block update reorients the present state.

ADS

The first 256 feed-forward units receive an additional adaptive feature path. ADS is independent of relation steering and supplies a compact specialized subspace inside the wider feed-forward computation.

Future Cone auxiliary objective

The primary objective is ordinary causal next-token prediction. During training, three lightweight predictor heads also encourage the current hidden state to retain information useful for near, middle, and farther future horizons. Projected future targets are detached; gradients flow through the current representation and predictor heads, not backward through future token states. The auxiliary weights are 0.02, 0.02, and 0.01, with a 40M-token ramp.

The Future Cone heads are training-only. They do not expose future tokens during inference and are excluded from the exported inference weights.

Configuration

Component	Value
Parameters	116,647,315
Physical layers	11
Hidden size	768
FFN size	2,752
Vocabulary	16,384
Maximum context	1,024
Query heads / KV heads	8 / 2
Attention head dimension	64
Local attention window	256
Global attention layers	3, 7, 10
GatedDeltaNet heads	6
Memory head dimension	48
Relation modes	12
Relation observer dimension	48
Deep Present loops	2
ADS units	256
Activation	SwiGLU
Normalization	RMSNorm / UnitRMS
Position encoding	RoPE, θ = 10,000
Dropout	0.0

Training data

ARO was trained only on the official detoxified English BabyLM 2026 Strict corpus at revision 9e57baaaa91ac3c638746be14d1d5fa6c789f4cf. The packed corpus contains exactly 100,000,000 whitespace-delimited words and 151,489,032 tokenizer tokens per pass.

Source	Words	Subword tokens
CHILDES	28,410,878	49,299,514
Project Gutenberg	25,576,896	34,568,431
OpenSubtitles	22,828,747	33,106,733
Simple English Wikipedia	15,314,317	24,223,950
BNC Spoken	7,620,671	9,926,687
Switchboard	248,491	363,717
Total	100,000,000	151,489,032

Documents were packed in a deterministic sorted-source stream as BOS + source tokens + EOS. Both tokenizer training and model training used the released BabyLM training corpus exclusively.

Tokenizer

The tokenizer is a 16,384-entry byte-level BPE trained from scratch on the official training split. It follows the BabyLM 2026 baseline tokenizer family while using an ARO-native vocabulary size.

Token	ID
<unk>	0
<s>	1
</s>	2
<pad>	3
<mask>	4

Tokenizer SHA-256:

8edfa792838717e0940c309880a71acf1b276d34981576212b0fffaf3cf1d909

Training recipe

Setting	Value
Objective	causal next-token CE + training-only Future Cone
Optimizer	AdamW
Betas	(0.9, 0.95)
Epsilon	1e-8
Weight decay	0.1
Gradient clipping	1.0
Peak learning rate	1e-3
Global tokens per optimizer step	131,072
Precision	BF16 autocast, TF32 enabled
Seed	1337
Optimizer steps	11,550
Hardware	one NVIDIA GeForce RTX 3090

The sequence-length curriculum kept the optimization batch constant while increasing context:

Word-exposure interval	Sequence length	Micro-batch	Gradient accumulation
0–600M	256	16	32
600–800M	512	8	32
800M–1B	1,024	4	32

The learning-rate schedule was transferred from earlier ARO learning-curve evidence. It used a 40M-subword warmup, a 1e-3 plateau, gradual half-cosine transitions toward 8e-4 and 6e-4, and a terminal learning rate of 6e-4.

The idea: the Ultimate Present and dependent origination

The starting image for ARO was a description of time encountered in accounts of near-death experience: the observer stands at the present, the past extends behind, and the future opens ahead. Past, present, and future appear as one panorama centered on a single point of observation. “The Ultimate Present” is the name given to that center.

ARO keeps one full-rank Present coordinate at the center of the model. Causal attention retrieves precise visible evidence. GatedDeltaNet maintains a compressed history. Depth routing recovers useful earlier transformations. Relation-conditioned feed-forward computation asks how these sources agree, differ, and interact. The Future Cone objective shapes the current representation toward several prediction horizons during training.

Buddhist dependent origination supplied a second principle: the present is formed through relations among conditions. ARO's twelve learned relation modes take their width from the twelve links of dependent origination, while a shared lower-triangular operator and its transpose provide forward and reverse readings. The modes are learned latent coordinates; they are not hard-coded labels for the twelve Buddhist links.

Deep Present brings the two sources together. Its first pass forms the current state from past evidence. Its second pass holds the principal past-side evidence fixed, regenerates the query from the updated Present, and reads the same past again. A changed Present extracts a new interpretation from fixed past evidence.

The philosophical sources provided a coherent architectural question. ARO converts that question into executable mechanisms whose language-model performance can be measured experimentally.

Usage

This model uses custom Transformers code. Review the repository source before enabling remote code.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "AwakeningOS/ARO-BabyLM-1B"

tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

inputs = tokenizer("Language is", return_tensors="pt").to(model.device)
with torch.no_grad():
    output = model.generate(
        **inputs,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Integrity and provenance

Artifact	SHA-256
Final native checkpoint	2cefd8615ad1aeec9bdad3857d1f658a0e8239433f921dee83e00dac8dfc81f9
Exported model.safetensors	6f6ee2e00d90fb4527f8c317a8b39948aa01d773f088ff24a44e04fff2f785ce
Tokenizer JSON	8edfa792838717e0940c309880a71acf1b276d34981576212b0fffaf3cf1d909
Packed token stream	bef1dc6b38cf9ed9d18fbb0e300134b1541f54a7c21498e0966614b4ec58148c
Frozen training source	95d3e4547d9cf3cef0097a4e83e7920e57720e4d3f820615b0d9d91fe0a3e7a0

The Hugging Face adapter saves deterministic RoPE and causal-mask buffers because Transformers meta-device loading leaves custom non-persistent buffers uninitialized. The exported weight tensors exactly preserve the trained parameters.

Intended use

ARO-BabyLM-1B is a research base model for studying data-efficient language acquisition, small causal architectures, grammatical generalization, and learning curves under a fixed exposure budget. It has not been instruction-tuned, preference-tuned, or safety-tuned.

Limitations

• The model is English-only and trained on a small, developmentally motivated corpus.
• It is a base next-token model, not an instruction-following assistant.
• The current reported BabyLM result is from one seed and an intermediate fast checkpoint suite.
• Strong BLiMP performance does not by itself establish broad reasoning, factual knowledge, or generation quality.
• The architecture contains several interacting mechanisms; the present run measures the complete system and does not isolate the causal contribution of each mechanism.
• Training data can contain social, cultural, and historical biases inherited from dialogue, subtitles, books, and Wikipedia-derived text.

Evaluation references

• BabyLM 2026 evaluation pipeline
• BabyLM 2025 findings
• BabyLM 2024 evaluation pipeline
• BabyLM 2026 Strict dataset

License

ARO-BabyLM-1B is released under the Apache License 2.0.

Copyright 2026 AwakeningOS.

Citation

A formal technical report and citation entry will be added after the complete evaluation package is frozen.