Prannesshkva/QU-SSM-15M

⚛️ QU-SSM: Gated Quasi-Unitary State Space Model (15M)

QU-SSM-15M is a standalone, from-scratch State Space Model architecture that eliminates contractive Hurwitz dissipation in linear recurrent models by parameterizing transitions in the Lie Group SO(N) via the Cayley transform while enabling selective noise filtering via dynamic forget gating.

• Author: Prannessh K.V.A. (prannesshkva@gmail.com)
• Official CERN Zenodo Record & DOI: 10.5281/zenodo.22177118
• Interactive Studio: Prannesshkva/QU-SSM-Studio
• License: Business Source License 1.1 (BSL 1.1)

---

⚡ Empirical Specifications & Pretraining Metrics

Architectural Metric	Measured Value	Description
Total Parameter Count	16,907,872 (~16.91M)	Clean standalone parameter footprint
Layer Configuration	6 Layers	d_model = 256, d_state = 16, d_ff = 768
Recurrence Operator	Gated Quasi-Unitary SO(N)	||U|| = 1.00000 (Zero contractive dissipation)
Pretraining Epochs & Steps	7,000 Gradient Steps (3 Epochs)	Pretrained from scratch on Tesla P100 GPU
Training Loss Drop	216.61 → 4.6196	97.9% Cross-Entropy Loss Reduction
Peak GPU Throughput	24,000 tokens / sec	Broadcasted 1D FFT Parallel Scan

---

📊 Benchmark 1: Multi-Depth Needle-In-A-Haystack (NIAH) Matrix

Empirical state norm stability measured on NVIDIA Tesla P100 GPU across sequence lengths from 1,024 to 16,384 tokens at 5 needle depths (10%, 30%, 50%, 70%, 90%):

Context Length	Insertion Depths Tested	Insertion Positions	State Norm (||h||)	Status
1,024 tokens	10%, 30%, 50%, 70%, 90%	102 to 921 tokens	4.7699	Stable (No Norm Collapse)
2,048 tokens	10%, 30%, 50%, 70%, 90%	204 to 1,843 tokens	5.2679	Stable (No Norm Collapse)
4,096 tokens	10%, 30%, 50%, 70%, 90%	409 to 3,686 tokens	4.7914	Stable (No Norm Collapse)
8,192 tokens	10%, 30%, 50%, 70%, 90%	819 to 7,372 tokens	4.7578	Stable (No Norm Collapse)
16,384 tokens	10%, 30%, 50%, 70%, 90%	1,638 to 14,745 tokens	4.8636	Stable (No Norm Collapse)

Observation: In standard Hurwitz-decay SSMs, the recurrent state norm decays exponentially toward zero as sequence length increases. In QU-SSM, the Lie-algebra SO(N) unitary rotation keeps the state norm strictly bounded between 4.76 and 5.26 across all depths up to 16,384 tokens.

---

🔬 Benchmark 2: Cross-Entropy Loss & Domain Generalization

Evaluated on 25,500 real tokens across 100 non-overlapping evaluation windows:

Evaluation Split	Dataset Type	Cross-Entropy NLL	Perplexity (PPL)
In-Distribution Validation	Synthetic Conversational & Educational	4.6196 NLL	101.5 PPL
Out-of-Distribution (WikiText-2)	Adult Encyclopedic Text (Test Split)	9.0166 NLL	8,238.5 PPL

Note: In accordance with neural scaling laws, sub-30M parameter models trained on synthetic/grade-school text exhibit high cross-entropy on adult encyclopedic vocabulary.

---

🔬 Mathematical Formulation

1. Lie-Algebra Skew-Symmetric Generator

$$\mathcal{H}=W-W^{\mathrm{\top }}\in \mathfrak{s}\mathfrak{o}(N)\backslash mathcal\{H\}\; =\; W\; -\; W^\backslash top\; \backslash in\; \backslash mathfrak\{so\}(N)$$

2. Cayley Unitary Operator ($|\mathcal{U}|_2 = 1.00000$)

$$\mathcal{U}={(I-\frac{1}{2}\mathcal{H})}^{-1}(I+\frac{1}{2}\mathcal{H})\in \text{SO}(N)\backslash mathcal\{U\}\; =\; \backslash left(I\; -\; \backslash frac\{1\}\{2\}\backslash mathcal\{H\}\backslash right)^\{-1\}\; \backslash left(I\; +\; \backslash frac\{1\}\{2\}\backslash mathcal\{H\}\backslash right)\; \backslash in\; \backslash text\{SO\}(N)$$

3. Gated Quasi-Unitary Recurrence

$$h_t={\gamma }_t\cdot ({\mathcal{U}}_t{h}_{t-1})+\sigma ({\mathrm{\Delta }}_t)\cdot (x_t\otimes B_t),{\gamma }_t=\sigma (\text{gate}(x_t))\in (0,1]h\_t\; =\; \backslash gamma\_t\; \backslash cdot\; (\backslash mathcal\{U\}\_t\; h\_\{t-1\})\; +\; \backslash sigma(\backslash Delta\_t)\; \backslash cdot\; (x\_t\; \backslash otimes\; B\_t),\; \backslash quad\; \backslash gamma\_t\; =\; \backslash sigma(\backslash text\{gate\}(x\_t))\; \backslash in\; (0,\; 1]$$

4. 0-Loop 1D Causal FFT Parallel Scan

$$h=\text{iFFT}(\text{FFT}(\mathcal{K})\odot \text{FFT}(u))\text{in }\mathcal{O}(L\log L)h\; =\; \backslash text\{iFFT\}\backslash left(\backslash text\{FFT\}(\backslash mathcal\{K\})\; \backslash odot\; \backslash text\{FFT\}(u)\backslash right)\; \backslash quad\; \backslash text\{in\; \}\; \backslash mathcal\{O\}(L\; \backslash log\; L)$$

---

🚀 Quickstart Inference (1-Line Hugging Face Loading)

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "Prannesshkva/QU-SSM-15M"

# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id, clean_up_tokenization_spaces=False)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)

prompt = "Photosynthesis is a process where plants use sunlight to"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids

# Autoregressive generation
generated = input_ids.clone()
for _ in range(45):
    with torch.no_grad():
        logits = model(generated).logits
        next_tok = torch.argmax(logits[:, -1, :], dim=-1, keepdim=True)
        generated = torch.cat([generated, next_tok], dim=-1)

output = tokenizer.decode(generated[0], skip_special_tokens=True, clean_up_tokenization_spaces=False)
print("QU-SSM OUTPUT:")
print(output)

---

📜 Citation & Intellectual Property

@software{qu_ssm_2026,
  author       = {Prannessh, K.V.A.},
  title        = {QU-SSM: Gated Quasi-Unitary State Space Models via Lie-Group SO(N) Evolution},
  month        = aug,
  year         = 2026,
  publisher    = {Zenodo},
  doi          = {10.5281/zenodo.22177118},
  url          = {https://doi.org/10.5281/zenodo.22177118}
}

Copyright © 2026 Prannessh K.V.A. Licensed under Business Source License 1.1 (BSL 1.1).