GRT-10M-Base (NeurIPS Submission Pretrained Checkpoint)

Note: This repository preserves the original, untouched 8.7M-parameter (2.2M non-embedding backbone) GRT-10M base checkpoint trained with first-generation PairAdam (336,000 steps on Nemotron) for our NeurIPS submission, prior to SymplecticAdam / HuygensSymplecticAdam curriculum fine-tuning. For the curriculum-aligned comparison, see MatterWaveAI/grt-10m.

A Geometric Rotation Transformer pretrained on Nemotron. GRT replaces the dense linear projections of a standard transformer with cascades of learned Givens rotations on SO(D), so the residual path contains no nn.Linear at all.

This is a research proof-of-concept at small scale. It is published to demonstrate that the architecture trains stably and evaluates end-to-end, not as a capable language model.

Model Details

Total parameters	8,685,283 (8.7M)
Non-embedding parameters	2,235,283 (2.2M)
d_model	256
n_blocks	6
n_heads	4
n_qkv_rotations	5
ffn_n_rotations	9
Vocabulary	50,000 (BPE)
Max sequence length	1,024
Precision	float32

The embedding dominates the parameter count: 6.45M of 8.7M parameters are the token embedding (50,000 tokens x 128 phase angles, plus a log-magnitude per token).

Training

Data	Nemotron
Steps	336,000
Optimizer	PairAdam
Peak LR	1e-3, cosine decay to 10%
Training loss	2.144

Metric	Value
Validation loss	4.033
Validation perplexity	56.4
Validation accuracy	0.400

Benchmarks (Pre-Fine-Tune Zero-Shot & Standard PairAdam Baseline)

Evaluated via grt-slm-bench (training/slm_bench.py) across 9 standard multiple-choice benchmarks (13,144 held-out examples) and GSM8K held-out perplexity:

Task (SLM-Bench)	Split (n)	Random	GRT-10M-Base (Zero-Shot)	Standard PairAdam Fine-Tune
BoolQ (google/boolq)	val (3,270)	50.00%	62.14%	61.13%
QNLI (nyu-mll/glue)	val (1,000)	50.00%	52.70%	49.70%
PIQA (baber/piqa)	val (1,838)	50.00%	52.18%	50.82%
WinoGrande (allenai/winogrande)	val (1,267)	50.00%	50.51%	50.75%
OpenBookQA (allenai/openbookqa)	test (500)	25.00%	30.00%	22.60%
ARC-Easy (allenai/ai2_arc)	test (2,376)	25.00%	26.94%	26.60%
HellaSwag (Rowan/hellaswag)	val (1,000)	25.00%	25.60%	29.10%
ARC-Challenge (allenai/ai2_arc)	test (1,172)	25.00%	22.53%	24.91%
CommonsenseQA (tau/commonsense_qa)	val (1,221)	20.00%	20.15%	22.85%
9-Task Multiple-Choice Macro Average	13,144 total	35.56%	38.08%	37.61%

Held-Out Math Perplexity	Split (n)	GRT-10M-Base (Zero-Shot)	Standard PairAdam Fine-Tune
GSM8K (openai/gsm8k)	test (500)	145.85 PPL	37.27 PPL

Reproduce with:

python -m training.slm_bench --checkpoint ./ --tasks boolq,arc_easy --device cuda

Zero-shot, via lm-evaluation-harness:

lm_eval --model hf --model_args pretrained=MatterWaveAI/grt-10m-base \
    --tasks boolq,arc_easy --batch_size 64

Usage

pip install geometric-rotation-transformer

from grt import GRTForCausalLM, RotationalTokenizerBPE

model = GRTForCausalLM.from_pretrained("MatterWaveAI/grt-10m-base")
tokenizer = RotationalTokenizerBPE.from_pretrained("MatterWaveAI/grt-10m-base")

input_ids = tokenizer("The geometry of language is", return_tensors="pt").input_ids
output = model.generate(input_ids, max_new_tokens=50, temperature=0.7)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Importing grt registers the architecture with the transformers Auto classes:

import grt  # registers model_type "grt"
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("MatterWaveAI/grt-10m-base")

The package is pure PyTorch and runs on CPU. Optimized CUDA kernels are optional and load automatically when installed.

trust_remote_code=True is not supported for this repository. The model code is distributed as a pip package rather than as Hub remote code, because the implementation spans a package with absolute imports that a flat Hub snapshot cannot resolve. Install geometric-rotation-transformer first; import grt then registers model_type: "grt" so the Auto classes resolve these weights.

Architecture

GRT replaces the standard Q/K/V projections and FFN weight matrices with cascades of learned 2x2 Givens rotations in a butterfly pattern. Each stage is modulated by data-dependent Clifford features:

feat  = [S0, V1, V2, B12]           # scalar energy, direction, cross-pair bivector
theta = angles + tanh(modulation)   # base angle + content-dependent nudge
r0    = x0*cos(theta) - x1*sin(theta)
r1    = x0*sin(theta) + x1*cos(theta)

Components:

• RotationalEmbedding — tokens stored as phase angles on the unit hypersphere
• RotationalAttention — Q/K/V via Givens cascades
• RotationalFFN — feed-forward via meta-coupled rotation stages
• PearlNorm — pair-radius normalization preserving per-pair phase
• RadialGELU — activation gating pair magnitudes, preserving direction
• PairAdam — AdamW with factored second moments: 1D params (angles, scales) use standard diagonal moments, 2D params (coupling matrices) use a row factor [m,1] and col factor [1,n] reconstructed as an outer product, at O(m+n) instead of O(mn). Optionally clamps per-step angular updates.

Because every projection is a rotation, the transform is norm-preserving by construction rather than by normalization.

Intended Use & Limitations

Intended for research into structured/rotational architectures. Not intended for production use or deployment.

Known limitations:

• English only, trained on a single corpus (Nemotron), so it inherits that corpus's coverage gaps and biases.
• No instruction tuning. This is a base model trained on next-token prediction.
• At 8.7M parameters, generated text is frequently incoherent. Do not expect instruction-following, factual reliability, or coherent long-form output.
• No safety tuning, RLHF, or content filtering of any kind.
• Context is capped at 1,024 tokens.

Citation

@article{grt2026,
  title={Geometric Rotation Transformer: Structured SO(n) Rotations for Efficient Sequence Modeling},
  author={MatterWave AI},
  year={2026},
  url={https://github.com/matterwaveai/grt}
}

License

Apache 2.0