GRT-30M-LoRA (MatterWaveAI/grt-30m-lora)

GRT-30M-LoRA is the canonical Toroidal Low-Rank Adaptation (GRTRotationalLoRA) release for MatterWaveAI/grt-30m (29,305,498 base parameters, 0 nn.Linear layers, 0 nn.Conv1d layers).

Because every projection in a Geometric Rotation Transformer is parameterized by phase angles on the flat torus $\mathbb{T}^P = (S^1)^P$ and radial scales rather than dense Euclidean weight matrices, standard linear LoRA ($W + BA$) cannot be applied without violating orthogonal/unitary manifold invariants. GRTRotationalLoRA instead attaches low-rank phase-shift matrices lora_A and lora_B directly to the butterfly rotation angle tensors:

$$\theta =({\theta }_{\text{base}}+\frac{\alpha }{r}AB)2\pi \backslash theta\; =\; \backslash left(\backslash theta\_\{\backslash text\{base\}\}\; +\; \backslash frac\{\backslash alpha\}\{r\}\backslash ,\; A\; B\backslash right)\; \backslash bmod\; 2\backslash pi$$

At inference time, merge_grt_lora(model) folds $\Delta\theta = \frac{\alpha}{r} A B$ directly into $\theta_{\text{base}}$ with bit-exact equivalence (max_abs_diff = 0.0) and zero runtime latency or parameter overhead.

---

Key Highlights & Metrics

Metric	Value
Adapter Class	GRTRotationalLoRA (transformers.models.grt.lora_grt)
Base Checkpoint	MatterWaveAI/grt-30m (29,305,498 params)
LoRA Rank ($r$) / Alpha ($\alpha$)	r = 8, lora_alpha = 16.0 (scaling = 2.0)
Trainable LoRA + Scale Parameters	140,741 (0.48% of base model)
Adapter Checkpoint Size	573.3 KB (grt_lora_adapter.safetensors)
nn.Linear / nn.Conv1d Layers	0 / 0 (100% Rotational Purity)
Optimizer	HessianPairAdam (Sophia-Torus analytic Gauss-Newton curvature)
Domain Assistant Loss (Before $\rightarrow$ After)	6.1720 $\rightarrow$ 4.7464
Domain Perplexity (Before $\rightarrow$ After)	479.14 $\rightarrow$ 115.16
Token Accuracy (Before $\rightarrow$ After)	19.5% $\rightarrow$ 28.8%
Pre-Merge vs. Post-Merge max_abs_diff	0.0e+00 (bit-exact)

---

Quickstart (transformers)

1. Load & Merge MatterWaveAI/grt-30m-lora into MatterWaveAI/grt-30m

import torch
from huggingface_hub import snapshot_download
from transformers import AutoTokenizer
from transformers.models.grt import GRTForCausalLM, load_grt_lora_pretrained

base_id = "MatterWaveAI/grt-30m"
adapter_dir = snapshot_download("MatterWaveAI/grt-30m-lora")

tokenizer = AutoTokenizer.from_pretrained(base_id)
model = GRTForCausalLM.from_pretrained(base_id, torch_dtype=torch.float32)

# Attach and fold the toroidal LoRA phase shifts into base_angles (0 inference overhead)
load_grt_lora_pretrained(model, adapter_dir, merge=True)

prompt = "<|im_start|>user\nExplain how GRTRotationalLoRA adapts a Geometric Rotation Transformer without linear layers.<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
out_ids = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(out_ids[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

2. Train Your Own Toroidal LoRA Adapter

from transformers.models.grt import (
    GRTForCausalLM,
    attach_grt_lora,
    build_grt_optimizer,
    merge_grt_lora,
    save_grt_lora_pretrained,
)

model = GRTForCausalLM.from_pretrained("MatterWaveAI/grt-30m")
attach_grt_lora(model, r=8, lora_alpha=16.0, tune_scales=True)

optimizer = build_grt_optimizer(model, lr=1e-2, weight_decay=0.0, hessian_mode=True)
# ... training loop ...
save_grt_lora_pretrained(model, "./my-grt-lora")
merge_grt_lora(model)  # zero-overhead merged inference

---

Complete MatterWaveAI/grt-30m* Research Suite on Hugging Face

1. Base Causal LM: MatterWaveAI/grt-30m
2. Chat Fine-Tune: MatterWaveAI/grt-30m-chat
3. 4-Expert Mixture-of-Rotations (MoR): MatterWaveAI/grt-30m-mor
4. Circular-Pooled Sentence Embedding: MatterWaveAI/grt-30m-embedding
5. Cross-Encoder Reranker: MatterWaveAI/grt-30m-reranker
6. Vision-Language (GRTVL): MatterWaveAI/grt-30m-vl
7. Canonical Tokenizer & Polar Embedding Bank: MatterWaveAI/grt-30m-tokenizer
8. Rotational Text-to-Speech (GRTTTS): MatterWaveAI/grt-30m-tts
9. Multimodal Vision-Audio-Language-Speech (GRTOmni): MatterWaveAI/grt-30m-omni
10. Toroidal Rotational LoRA (GRTRotationalLoRA): MatterWaveAI/grt-30m-lora
11. Rotational Speech-to-Text ASR (GRTASR): MatterWaveAI/grt-30m-asr

• Whitepaper: The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (GRT_Whitepaper.pdf)

---

License

Apache 2.0

---

Citation

@techreport{householter2026grt,
  title       = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
  author      = {Householter, David},
  institution = {MatterWave AI},
  year        = {2026},
  url         = {https://matterwave.ai}
}