Instructions to use AndrewThompson1233/maba-v2-architecture with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AndrewThompson1233/maba-v2-architecture with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="AndrewThompson1233/maba-v2-architecture")

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("AndrewThompson1233/maba-v2-architecture", device_map="auto")

RWKV

How to use AndrewThompson1233/maba-v2-architecture with RWKV:

# No code snippets available yet for this library.

# To use this model, check the repository files and the library's documentation.

# Want to help? PRs adding snippets are welcome at:
# https://github.com/huggingface/huggingface.js

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use AndrewThompson1233/maba-v2-architecture with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AndrewThompson1233/maba-v2-architecture"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AndrewThompson1233/maba-v2-architecture",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/AndrewThompson1233/maba-v2-architecture

SGLang

How to use AndrewThompson1233/maba-v2-architecture with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "AndrewThompson1233/maba-v2-architecture" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AndrewThompson1233/maba-v2-architecture",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "AndrewThompson1233/maba-v2-architecture" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AndrewThompson1233/maba-v2-architecture",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use AndrewThompson1233/maba-v2-architecture with Docker Model Runner:
```
docker model run hf.co/AndrewThompson1233/maba-v2-architecture
```

Maba v2 Architecture

Linear Recurrence & Sparse Attention Hybrid Architecture

Production Release of the Experimental Maba v1.5 Prototype

Maba v2 is the official production release and hardened evolution of the experimental Maba v1.5-exp prototype (GitHub).

This release provides a fully verified, stable, and substantially improved architecture:

Autograd & Dispatcher Fixes: Completely resolved backward gradient propagation edge cases and fallback dispatcher issues that existed in v1.5 experimental builds.

Million-Token Context Scaling: Verified scaling to 1,000,000+ tokens with 39.6x KV-cache reduction (1.20 GB for 1M tokens in FP16) and 100% fine-grained retrieval (Rank #1 out of 15,625 blocks).

Strict O(1) Decoding: Constant 35–37 ms/token generation latency on consumer GPUs with zero sequence-length slowdown.

NoPE Positional Stability: Replaces RoPE with recurrent exponential decay (α_t) to ensure temporal invariance without frequency phase drift.

Overview

Maba v2 Architecture (maba-v2-architecture) is a reference PyTorch implementation of a 3:1 hybrid architecture uniting linear recurrence (DGDA) and sparse global attention (MABA-SA).

Traditional dense transformers suffer from O(L²) prefill memory and O(L) linear decode slowdown. Pure linear recurrent models struggle with associative recall across distant context. Maba solves this dilemma by routing 75% of compute through constant-state recurrence and 25% through latent-compressed sparse attention with anti-dilution centroid routing.

Strict O(1) Decode Latency: 35–37 ms/token flat up to 1,000,000 tokens on consumer GPUs.
39.6x KV-Cache Compression: 1.20 GB for 1M tokens in FP16 (vs 48.8 GB for dense attention).
1,000,000 Token Fact Extraction: Single-needle retrieval at Token #742,189 with Rank #1 out of 15,625 blocks and 100% fine-grained attention focus.
NoPE Temporal Invariance: Replaces RoPE with exponential recurrent decay (α_t) to prevent frequency phase distortion over long distances.

Frontier Architectural Comparison

Architectural Comparison: Maba vs Qwen3.8-Flash-Next vs MiniCPM-5 vs Dense Transformer

Architecture	Macro-Topology	Attention Paradigm	Decode Complexity	KV Cache @ 131k	KV Cache @ 1M	Max Context
Maba (Canonical)	3:1 Hybrid (15 DGDA : 5 MABA-SA)	Latent MLA (d_c=128) + 64:1 Centroids	O(1) Flat (35 ms)	163.6 MB	1.20 GB	1,000,000+ (NoPE)
Qwen3.8-Flash-Next	Hybrid GDN + QSA MoE (6B active)	Micro-Block Sparse Attention	Sublinear O(log L)	640.0 MB	4.80 GB	262k / 1M (YaRN)
MiniCPM-5	Dense CausalLM (1B / 2B)	100% Dense GQA	Linear O(L)	3.20 GB	24.50 GB	131,072 (RoPE)
Dense Transformer	Standard Transformer	100% Dense Softmax MHA	Linear O(L)	6.40 GB	48.82 GB	64,000 max (OOM)

Scaling & Resource Specifications

All parameter scaling laws, model tiers (100M to 30B), analytical KV-cache memory formulations, and empirical context scaling comparisons are documented in SCALING.md.

Architectural Specifications

Reference Config: config.json (101,282,319 parameters).
Macro-Stack (20 layers, 3:1 ratio):
- 15 layers: DGDA (Decoupled Gated Delta Attention) linear recurrence.
- 5 layers: MABA-SA (Sparse Attention) with MLA latent compression (d_c = 128).
Factorized Embeddings: Vocab 32,768 -> 128 -> 640 in maba_sparse/model.py.
Anti-Dilution Routing: DG-Indexer uses hybrid pooling 0.5 × (mean + max) with distance decay penalty λ · log(1 + Δ) to protect salient single-token facts against background noise.
Superposition Attention: 3 streams dynamically superposed via data-dependent gate logits:
1. Local Sliding Window (128 tokens + 4 sinks).
2. Sparse Top-32 Blocks (2,048 gathered tokens).
3. Hierarchical Context Attention (HCA 64:1 compressed prefix).
Multi-Backend Kernels: maba_sparse/kernels/ provides fused Triton GPU kernels (264k–375k tok/s on Tesla T4), CPU OpenMP parallel kernels, and PyTorch autograd fallbacks.

Quick Start

Installation

git clone https://github.com/AndrewThompson1233/maba-v2-architecture.git
cd maba-v2-architecture
pip install -e .

Autoregressive Generation

import torch
from maba_sparse.model import MabaSparseForCausalLM, get_101m_config

config = get_101m_config()
model = MabaSparseForCausalLM(config).cuda().eval()

input_ids = torch.tensor([[101, 2045, 312]], device="cuda")
with torch.no_grad():
    output = model.generate(input_ids, max_new_tokens=32, temperature=0.7)
print(output)

Training

# Distributed Data Parallel training on multi-GPU
torchrun --nproc_per_node=2 train.py \
    --model maba_sparse \
    --dataset synthetic \
    --steps 100 \
    --batch_size 4 \
    --fp16

Benchmark Suite

All benchmark suites are consolidated in benchmark.py. Full hardware metrics on Tesla T4 GPUs are recorded in BENCHMARK_REPORT.md and benchmark_results.json.

# Frontier architectural comparison vs Qwen3.8-Flash-Next & MiniCPM-5
python benchmark.py --mode arch

# End-to-end model & attention scaling (Maba vs Dense Transformer)
python benchmark.py --mode model --contexts 128,256,512,1024,2048,4096

# Constant O(1) decode latency scaling
python benchmark.py --mode decode

# KV-cache footprint comparison (40x reduction)
python benchmark.py --mode memory

# 1,000,000 token single-needle fact extraction
python benchmark.py --mode needle

# 50 Hard Negatives ('semantic mines') & Multi-Hop reasoning across 640k tokens
python benchmark.py --mode multihop

# Triton hardware kernel throughput
python benchmark.py --mode triton

# Run all suites sequentially
python benchmark.py --mode all

Test Suite

Run the full automated unit test suite (649 tests):

pytest -q

All 23 test modules in tests/ verify causal masking, autograd graph integrity, memory invariance, numerical stability, and hardware kernel parity.

License & Attribution

Maba v2 Architecture is released under the MABA Open Architecture License (MOAL-1.0).

Author: Andrew Thompson (AndrewThompson1233)
Commercial & Research Use: Permitted without royalty fees.
Attribution: Any derivative architecture, implementation, checkpoint, or paper must state:

Created based on Maba v2 Architecture by Andrew Thompson
Anti-Plagiarism Protection: The name Maba / Maba v2 and its foundational mechanisms (DGDA, MABA-SA, DG-Indexer, HCA) may not be renamed, rebranded, or claimed under different names when adapting or copying this architecture.

See LICENSE for the full license text.

Downloads last month: -