ROTOR-50M-MoE: Sparse Mixture-of-Experts Geometric Sequence Model

ROTOR-50M-MoE is the flagship Sparse Mixture-of-Experts (MoE) foundation sequence model of the ROTOR (Rotational Orientation Token Organization & Recall) architecture, designed and invented by Prannessh K.V.A..

ROTOR-50M-MoE combines 48 Multi-Head 3D Quaternionic Rotor Frames (SU(2)) with 8 parallel SwiGLU Experts and Top-2 Routing, delivering the knowledge capacity of a 64M parameter model while activating only 17.97 Million parameters per token.

---

⚡ Specifications & Architecture Breakdown

• Total Parameters: 64,044,192 (64.04 Million)
• Active Compute per Token: 17,965,344 (17.97 Million Active - Top-2 Routing)
• Experts: 8 SwiGLU Experts with GShard auxiliary load balancing
• Layers: 6 Blocks
• Hidden Dimension: 384
• Rotor Heads: 48 independent 3D quaternion heads per layer
• Context State Footprint: 4.50 Kilobytes (Strictly Constant O(1))
• Time Complexity: Strictly Linear O(N + M)
• Target Domain: Multimodal speech recognition (Whisper alternative), high-capacity vision-language alignment, audio-text cross-encoding

---

🚀 Quickstart Usage

import torch
from modeling_rotor import RotorForConditionalGeneration
from configuration_rotor import RotorConfig

# Initialize ROTOR-50M-MoE Flagship Model
config = RotorConfig(
    vocab_size=8192,
    d_model=384,
    num_layers=6,
    num_heads=48,
    d_ffn=1024,
    use_moe=True,
    num_experts=8,
    num_experts_per_tok=2,
    moe_aux_loss_coef=0.01
)
model = RotorForConditionalGeneration(config)

# Forward pass with 500-token context stream & 16-token query stream
context_ids = torch.randint(0, 8192, (2, 500))
query_ids   = torch.randint(0, 8192, (2, 16))

outputs = model(input_ids=query_ids, context_input_ids=context_ids)
print("Logits Shape:", outputs.logits.shape) # [2, 16, 8192]

---

---

🎧🖼️ Multimodal Inference (Audio & Vision)

ROTOR natively supports direct continuous-stream cross-alignment without an attention matrix. You can feed raw audio waveforms or RGB image tensors directly:

import torch
from transformers import AutoModelForCausalLM
from multimodal_encoders import RotorAudioEncoder, RotorVisionEncoder

# 1. Load Pretrained ROTOR Model
model = AutoModelForCausalLM.from_pretrained('Prannesshkva/ROTOR-15M', trust_remote_code=True)

# 2. AUDIO-TO-TEXT (16kHz raw waveform)
audio_enc = RotorAudioEncoder(d_model=384)
waveform = torch.randn(1, 16000) # 1 sec audio
audio_tokens = audio_enc(waveform)

# Query text decoder
query_ids = torch.tensor([[50256]]) # BOS token
out_audio = model(input_ids=query_ids, context_embeds=audio_tokens)
print("Audio-to-Text Logits:", out_audio.logits.shape)

# 3. VISION-TO-TEXT (224x224 RGB image)
vision_enc = RotorVisionEncoder(d_model=384)
image = torch.randn(1, 3, 224, 224) # RGB image tensor
vision_tokens = vision_enc(image)

out_vision = model(input_ids=query_ids, context_embeds=vision_tokens)
print("Vision-to-Text Logits:", out_vision.logits.shape)

⚖️ Intellectual Property & Licensing

ROTOR-50M-MoE is protected under a multi-licensing framework:

1. GNU Affero General Public License v3 (AGPLv3): OSI-approved open-source copyleft protection.
2. PolyForm Noncommercial License 1.0.0: Frictionless academic and research safe harbor.
3. Commercial Enterprise License: Direct inquiries to: prannessh.kva@gmail.com.

Sole Architect & Inventor: Prannessh K.V.A. (@prannesshkva)