GRT-30M-Embedding (Circular-Pooled Rotational Sentence Encoder)

An embedding-only Geometric Rotation Transformer (GRTModel) contrastively fine-tuned from MatterWaveAI/grt-30m for semantic textual similarity and dense retrieval using planar-pair circular pooling (sentence-transformers).

Instead of dense linear projections (nn.Linear), GRT represents token embeddings in polar coordinates (phase_angles + log_magnitude) and transforms hidden states through cascades of content-modulated 2×2 Givens rotations on SO(D) interleaved with butterfly permutations. Because every hidden state is structured as D / 2 = 128 planar (x, y) coordinate pairs, circular pooling decomposes each pair into its unit direction (cos θ, sin θ) and radius r, averages the unit directions and radii across non-padding tokens, and recombines them into a 256-dimensional sentence embedding.

---

Model Details

Property	Value
Architecture	GRTModel (model_type: "grt", encoder/backbone without causal LM head)
Base model	MatterWaveAI/grt-30m
Pooling strategy	circular (Pooling(embedding_dimension=256, pooling_mode="circular") + Normalize)
Output dimensionality	256 (128 planar (x, y) pairs, L2-normalized)
Total stored parameters	29,305,497 (29.31M, lm_head.output_scale stripped)
Non-embedding backbone parameters	9,740,197 (9.74M across 12 rotational blocks)
hidden_size (d_model)	256
num_hidden_layers (n_blocks)	12
num_attention_heads (n_heads)	4 (head_dim = 64)
intermediate_size (d_ff)	1,024
n_qkv_rotations / n_mixer_rotations / ffn_n_rotations	11 / 1 / 21
Vocabulary	151,669 (Qwen 3 BPE)
Max sequence length	512 (supports up to 2,048 positions)

---

Two-Stage Contrastive Fine-Tuning Method (Intel Arc XPU)

Starting from the pretrained MatterWaveAI/grt-30m causal checkpoint (lm_head.output_scale stripped to form a headless GRTModel encoder), the 151,669 lexical embedding.phase_angles (19.41M parameters) were frozen to preserve the pretrained Qwen 3 polar token geometry while training the 9,740,196 rotational backbone parameters + 151,669 token log_magnitude weights (9,891,865 trainable parameters) in FP32 on an Intel Arc 140T GPU (xpu:0, 32 GB) using fused oneAPI DPC++ SYCL Givens and Clifford kernels (~75s total wall-clock time):

Stage	Dataset & Budget	Objective & Hyperparameters
Parameter Freezing	embedding.phase_angles frozen (19.41M params)	9,891,865 trainable parameters (12 rotational blocks + embedding.log_magnitude); max grad norm 1.0
Stage 1: AllNLI Contrastive Alignment	sentence-transformers/all-nli (triplet split) · 150 steps (4,800 triplets, <1% of corpus)	In-batch MultipleNegativesRankingLoss + explicit hard negatives (τ = 0.05), batch_size = 32, max_length = 96, PairAdam(lr=1.5e-4, coupling_lr_mult=3.0, max_angular_step=0.05)
Stage 2: STS-B Rank & Metric Calibration	sentence-transformers/stsb (train split, 5,749 pairs) · 2 epochs (358 steps)	CoSENT rank-consistency loss (τ = 20.0) + 2.0 × CosineMSE, batch_size = 32, max_length = 128, PairAdam(lr=8e-5, coupling_lr_mult=3.0, max_angular_step=0.05)

---

Evaluation Results (STS-B)

Evaluated on the STS Benchmark (sentence-transformers/stsb) validation and test splits using circular pooling and cosine similarity:

Split	Metric	GRT-30M (Pretrained Base)	GRT-30M-Embedding (PairAdam)	AdamW Ablation¹	Delta (PairAdam vs. Base)
STS-B Validation	Spearman (×100)	45.51	71.79	72.89	+26.28
STS-B Test	Spearman (×100)	28.49	62.42	63.17	+33.93
STS-B Test	Pearson (×100)	29.81	63.30	63.72	+33.49

¹ Optimizer comparison note: The published checkpoint is trained with PairAdam (zero weight decay and [-π, π] wrapping on rotation angles, max_angular_step = 0.05, and rank-1 factored second moments with a 3× learning-rate multiplier on Cl(2,0) coupling tensors) for consistency with GRT-30M pretraining. In a head-to-head run on the identical 508-step schedule and random seed (42), standard AdamW (1× coupling LR, full diagonal second moments) reaches 72.89 Val / 63.17 Test Spearman (+1.10 / +0.75 points), indicating that while PairAdam's factored moments and 3× coupling LR multiplier stabilize large-batch (524k tok/step) pretraining from scratch, standard diagonal moments at 1× coupling LR are slightly gentler during short, small-batch (B = 32) downstream fine-tuning.

---

Usage

1. sentence-transformers (Recommended)

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("MatterWaveAI/grt-30m-embedding")

sentences = [
    "Plants convert sunlight into chemical energy through photosynthesis.",
    "Chlorophyll absorbs light energy to synthesize glucose in plant leaves.",
    "The stock market experienced volatility during the afternoon session.",
]

embeddings = model.encode(sentences)
similarities = model.similarity(embeddings, embeddings)
print(similarities)

2. Hugging Face transformers (GRTModel + Circular Pooling)

import torch
import torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer

model_id = "MatterWaveAI/grt-30m-embedding"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id).eval()


def circular_pool(token_embeddings: torch.Tensor, attention_mask: torch.Tensor) -> torch.Tensor:
    pairs = token_embeddings.float().unflatten(-1, (-1, 2))
    mask = attention_mask.float().unsqueeze(-1).unsqueeze(-1)
    radii = pairs.norm(dim=-1, keepdim=True)
    directions = pairs / torch.clamp(radii, min=1e-9)
    denom = torch.clamp(mask.sum(dim=1), min=1e-9)
    pooled = ((directions * mask).sum(dim=1) / denom) * ((radii * mask).sum(dim=1) / denom)
    return F.normalize(pooled.flatten(start_dim=1), p=2, dim=-1).to(token_embeddings.dtype)


inputs = tokenizer(
    ["How does photosynthesis work?", "Plants convert sunlight into chemical energy."],
    padding=True,
    truncation=True,
    return_tensors="pt",
)
with torch.no_grad():
    outputs = model(**inputs)
    embeddings = circular_pool(outputs.last_hidden_state, inputs["attention_mask"])

print("Cosine similarity:", float((embeddings[0] @ embeddings[1]).item()))

---

Complete MatterWaveAI/grt-30m* Research Suite on Hugging Face

1. Base Causal LM: MatterWaveAI/grt-30m
2. Chat Fine-Tune: MatterWaveAI/grt-30m-chat
3. 4-Expert Mixture-of-Rotations (MoR): MatterWaveAI/grt-30m-mor
4. Circular-Pooled Sentence Embedding: MatterWaveAI/grt-30m-embedding
5. Cross-Encoder Reranker: MatterWaveAI/grt-30m-reranker
6. Vision-Language (GRTVL): MatterWaveAI/grt-30m-vl
7. Canonical Tokenizer & Polar Embedding Bank: MatterWaveAI/grt-30m-tokenizer

• Whitepaper: The Geometric Rotation Transformer (GRT): A Purely Rotational Foundation Architecture (GRT_Whitepaper.pdf)

---

License

Apache 2.0

---

Citation

@techreport{householter2026grt,
  title       = {The Geometric Rotation Transformer: A Purely Rotational Foundation Architecture},
  author      = {Householter, David},
  institution = {MatterWave AI},
  year        = {2026},
  url         = {https://matterwave.ai}
}