Ael-504M: A Sub-Billion Parameter Hybrid SSM-Attention Any-to-Any Multimodal Architecture

---

📌 Executive Abstract

Ael-504M is an original sub-billion parameter foundation model engineered for high-throughput, on-device multimodal reasoning and cross-sensory synthesis. Modern frontier architectures often force a compromise between the continuous linear efficiency of State Space Models (SSMs) and the quadratic expressive precision of Multi-Head Attention (MHA). Ael-504M unifies these paradigms into a cohesive 24-layer hybrid backbone, interleaving Mamba-2 State Space Duality (SSD) for long-context linear recurrence with Rotary Delta-Attention (RDA) for geometric focus and exact fact retrieval.

To maximize compute efficiency, each layer is coupled with an 8-Expert Centroid-Routed SwiGLU Mixture-of-Experts (MoE) network, activating only ~182 Million parameters per token while retaining the capacity of a half-billion parameter model. Integrated with an Any-to-Any Cross-Attention Fusion Hub, Ael-504M natively compresses visual patches, acoustic Mel-spectrograms, and textual tokens into a shared $1024$-dimensional latent manifold—all within a compact ~1.01 GB VRAM footprint.

---

🏛️ Architectural Innovations & Theoretical Foundations

graph TD
    subgraph "1. SENSORY ENCODERS"
        V["Vision (224x224 Conv2D Patches, k=16)"] --> FH["Any-to-Any Cross-Modal Fusion Hub"]
        A["Audio (80-Mel Spectrograms, Conv1D)"] --> FH
        T["Text (50,304 BPE Token Embeddings)"] --> FH
    end

    FH --> L["24 HYBRID INTERLEAVED BLOCKS"]

    subgraph "2. DUAL-BRAIN SEQUENCE MODELING"
        L --> SSD["Mamba-2 SSD Layers (Interval = 6, O(L) Infinite Linear Scan)"]
        L --> RDA["Rotary Delta Attention (RoPE-MHA, O(L^2) Geometric Focal Recall)"]
        SSD --> MOE["Centroid-Routed SwiGLU MoE (8 Experts, Top-2 Dispatch)"]
        RDA --> MOE
    end

    MOE --> OUT["UNIFIED GENERATION (Causal Text + Vision/Audio Heads)"]

---

1. Dual-Brain Hybrid Sequence Modeling ($O(L) + O(L^2)$)

Ael-504M solves the quadratic memory bottleneck of pure Transformers without sacrificing in-context associative recall:

• Mamba-2 State Space Duality (SSD) Layers (Cadence: Every 6th Layer): Models continuous sequence dynamics via structured state-space matrices ($N=64, \text{expand}=2$). Enables linear $O(L)$ time complexity and constant $O(1)$ memory state rollout during autoregressive inference.
• Rotary Delta-Attention (RDA) Layers (Interleaved Layers): Applies dense multi-head attention ($16$ heads, $d_k=64$) equipped with dynamic on-the-fly Rotary Positional Embeddings (RoPE). Guarantees rotational invariance and sharp token-to-token associative retrieval.

---

2. Contrastive Centroid Mixture-of-Experts (MoE)

Rather than standard feed-forward networks (FFN) that compute dense matrix multiplications across all parameters, every layer in Ael-504M incorporates an 8-expert sparse subnetwork:

• Cosine Centroid Routing: Token embeddings $x \in \mathbb{R}^D$ are routed using normalized cosine similarity against learnable semantic centroids $C \in \mathbb{R}^{E \times D}$ with dynamic temperature scaling: $$\text{Gate}(x) = \text{Softmax}\left(\frac{\cos(x, C)}{\tau}\right)$$
• Top-2 Sparse Dispatch: Only the top 2 highest-affinity experts are activated per token, reducing FLOPs by ~64% while preserving total representational capacity.
• Dual-Loss Auxiliary Regularization: Optimized with Frobenius centroid orthogonality penalties and Switch-Transformer load balancing to prevent expert collapse.

---

3. Native Any-to-Any Tri-Modal Latent Manifold

• Vision Projection: 2D convolution patch projection ($16 \times 16$ kernel) maps $224 \times 224$ images into dense visual feature tokens.
• Audio Projection: 1D convolution maps $80$-channel Mel-spectrograms into acoustic latent representations.
• Cross-Modal Fusion Hub: Computes bidirectional cross-attention with 32 learnable query tokens, projecting visual and acoustic data directly into the shared causal text stream.

---

📊 Comprehensive Technical Specifications

Parameter / Dimension	Value	Architectural Description
Total Parameter Count	504,429,976 (~0.504B)	Full structural parameter capacity
Active Parameters / Token	~182,000,000	Active parameters per inference step (via Top-2 MoE)
Total Layers	24 Blocks	Pre-norm residual blocks
SSM Cadence Interval	Every 6 Layers	Layers 0, 6, 12, 18 = Mamba-2 SSD; Others = RDA
Hidden Dimension ($D$)	1,024	Unified latent manifold dimension
Attention Heads	16 Heads ($d_k = 64$)	Multi-Head Attention with Dynamic RoPE
MoE Experts	8 Experts	Top-2 Dynamic Dispatch per token
Expert Hidden Dimension	696	SwiGLU inner intermediate dimension
SSM State Dimension ($N$)	64	Continuous State Space Memory dimension
Vocabulary Size	50,304	BPE Tokenizer with ChatML control tokens
Context Window	2,048 Tokens	Maximum positional context limit
VRAM Footprint (FP16/BF16)	~1.01 GB	Fully deployable on consumer laptops, edge robotics, and phones

---

🛡️ Intellectual Property, Trademark & Licensing Notice

⚖️ Business Source License 1.1 (BSL 1.1)

Copyright (c) 2026 Prannessh K.V.A. All Rights Reserved.

Trademarks: "Ael"™, "Ael-504M"™, "Ael-AI"™, and associated emblems are common law trademarks of Prannessh K.V.A.

Ael-504M is licensed under the Business Source License 1.1 (BSL 1.1):

• Free Use Grant: 100% free of charge for non-commercial research, personal projects, academic study, and non-production testing and evaluation.
• Commercial License Requirement: Any commercial deployment, hosted cloud API monetization, or commercial product integration requires a separate commercial license from the author.
• Change Date (Sunset Clause): On August 22, 2030, this version of the software automatically transitions to the Apache License, Version 2.0.
• Patent Defense: Any party initiating patent litigation against the author forfeits all rights under this license.

---

📄 Academic Research Paper & Technical Blueprint

The comprehensive academic manuscript ("Ael-504M: A Sub-Billion Parameter Hybrid Mamba-2 and Delta-Attention Any-to-Any Multimodal Architecture") containing complete mathematical proofs, scaling laws, and ablation studies is available upon formal request.

• 📧 Request Full Research Paper: Contact Author via Email
• 💼 Author Profile: Prannessh K.V.A. on Hugging Face

---

⚡ Quickstart & Inference

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Prannesshkva/Ael-504M"

# 1. Load Tokenizer and Ael-504M Model
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16 if torch.cuda.is_available() else torch.float32,
    device_map="auto",
)

# 2. Format Prompt with ChatML
prompt = "<|im_start|>user\nExplain the architecture of hybrid state-space models in simple terms.<|im_end|>\n<|im_start|>assistant\n"
input_ids = tokenizer.encode(prompt, return_tensors="pt").to(model.device)

# 3. Autoregressive Generation
output_ids = model.generate(
    text_ids=input_ids,
    max_new_tokens=128,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.15,
)

response = tokenizer.decode(output_ids[0][input_ids.shape[1]:], skip_special_tokens=True)
print(response)

---

📚 Formal Academic Citation

To cite the Ael-504M architecture or foundation model in academic research, please use the following BibTeX entry:

@article{prannessh2026ael504m,
  title={Ael-504M: A Sub-Billion Parameter Hybrid Mamba-2 and Delta-Attention Any-to-Any Multimodal Architecture},
  author={Prannessh, K. V. A.},
  journal={CERN Zenodo Technical Report},
  year={2026},
  doi={10.5281/zenodo.22055871},
  url={https://doi.org/10.5281/zenodo.22055871},
  note={Correspondence: prannesshkva@gmail.com}
}