ISOM-R1-Coder-16B-MoE: 160K Repository-Scale Code & Deliberative MoE

Overview

ISOM-R1-Coder-16B-MoE is a 16-billion parameter Mixture-of-Experts (MoE) code generation model featuring 64 routed experts (2 shared, TopK=6 activated per token) and Multi-Head Latent Attention (MLA). Built upon the DeepSeek-Coder-V2-Lite architecture, it is designed for bounded-state execution over massive 160K repository contexts.

Traditional Transformers suffer from quadratic memory scaling in long contexts. By projecting Multi-Head Latent Attention key-value states onto a continuous isometric manifold (Float64 Cayley SO(d)), ISOM constrains the active working memory footprint over long token sequences.

Model	Primary Architecture Role	Base Lineage (Independent Derivative)	Total / Active Parameters	Max Context	Cache Complexity	Hardware Target
ISOM-R1-Coder-16B-MoE	160K Bounded Code & MLA MoE	DeepSeek-Coder-V2-Lite (Non-Endorsed)	15.71B / 2.36B Active	163,840 (160K)	O(1) Bounded Manifold (Architectural Spec)	16GB Cloud / Multi-GPU
ISOM-R1-Enterprise-40B	40B System-2 Foundation Reasoning	Falcon-40B (Non-Endorsed)	40.0B Dense	32,768 (32K)	O(1) Bounded State (Architectural Spec)	Enterprise Multi-GPU (24GB-80GB)
ISOM-R1-Coder-1.5B-Instruct	128K Repository Code Intelligence	Qwen2.5-Coder-1.5B-Instruct (Non-Endorsed)	1.54B Dense	131,072 (128K)	O(1) Bounded State (Tesla T4 Verified)	8GB Developer Laptops / Edge
ISOM-R1-Reasoning-1.5B-Instruct	32K System-2 Mathematical Deliberation	Qwen2.5-1.5B-Instruct (Non-Endorsed)	1.54B Dense	32,768 (32K)	O(1) Bounded State (Tesla T4 Verified)	8GB Edge / Consumer GPUs
ISOM-R1-Edge-130M-MoE	Unbounded Recurrent Drafter & SSM	Standalone Continuous SSM + MoE	134.89M / 58.27M Active	Unbounded Recurrence	O(1) Recurrent State (0.0469 MB Verified)	Ultra-Low Power Edge & CPU

Theoretical Architectural Specifications

Metric	Specification
Total Parameters	15.71 Billion
Active Parameters / Token	2.36 Billion
Architecture	Multi-Head Latent Attention (MLA) + MoE (64 routed, 2 shared)
Base Model	deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct (Non-Endorsed)
Context Window	163,840 tokens (160K)
Reasoning Threshold (B)	16,384 tokens
Working Memory Complexity	O(1) Bounded State (Architectural Spec)
Weight Data Types	BFloat16 / Float16 (Native), 4-bit AWQ/bitsandbytes (Consumer Tier)

Mathematical Foundation: The 160K Memory Wall

In traditional Multi-Head Latent Attention without ISOM, cache memory grows linearly with context length:

$M_{\text{vanilla}}(N) = 2 \times L \times H \times d_v \times N \times \text{bytes}$

At 160K context (163,840 tokens) across 27 layers with 16 attention heads and $d_v = 128$: $M_{\text{vanilla}}(160\text{K}) = 2 \times 27 \times 16 \times 128 \times 163,840 \times 2 \approx \mathbf{36.21\text{ GB}}$

Under the ISOM architecture, historical key-value states exceeding the reasoning threshold $B = 16,384$ are projected onto the isometric manifold:

$M_{\text{ISOM}}(N) = \min(N, B) \times \text{stride} + \text{dim}(\mathcal{M}_{\text{Cayley}})$

Sequence Length	Vanilla Attention KV (FP16)	ISOM Latent State Spec (FP16)	ISOM Latent State Spec (INT8)
4,096 tokens	0.90 GB	0.90 GB	0.45 GB
16,384 tokens	3.62 GB	3.62 GB	1.81 GB
32,768 tokens	7.24 GB	3.62 GB	1.81 GB
65,536 tokens	14.48 GB	3.62 GB	1.81 GB
163,840 tokens (160K)	36.21 GB	3.62 GB	1.81 GB

Architectural Specification Notice: Values above represent theoretical dimensional derivations based on DeepSeek-Coder-V2-Lite Multi-Head Latent Attention (MLA) parameters. Empirical validation across 160K sequences requires an 80GB enterprise GPU cluster and is not claimed as an audited hardware measurement.

Quickstart & Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "Prannesshkva/ISOM-R1-Coder-16B-MoE"

# Load Tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# Load Model
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

prompt = """<|im_start|>user
Write a high-performance concurrent queue in Python using lock-free atomic CAS operations.<|im_end|>
<|im_start|>assistant
"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=512,
        temperature=0.3,
        do_sample=True
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation & Contact

@software{isom_coder_16b_2026,
  author = {Prannessh K.V.A.},
  title = {ISOM-R1-Coder-16B-MoE: 160K Repository-Scale Code & Deliberative MoE},
  year = {2026},
  publisher = {Zenodo},
  doi = {10.5281/zenodo.22649142},
  url = {https://doi.org/10.5281/zenodo.22649142}
}

Sole Author & Architect: Prannessh K.V.A.
LinkedIn: Prannessh K.V.A.
Zenodo DOI: 10.5281/zenodo.22649142
Space: ISOM-Benchmark

Notice of Non-Endorsement & Independent Lineage

Independent Derivative Work: ISOM-R1-Coder-16B-MoE is an independent research implementation developed solely by Prannessh K.V.A. (Author, Architect & IP Holder). It utilizes the model weights and architecture of deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct under the official DeepSeek Model License. This release is not endorsed by, sponsored by, or affiliated with DeepSeek AI or its parent entities. All modifications, continuous Cayley SO(d) projection operators, and isometric memory-bounding manifolds are original contributions of the author.

Downloads last month: 401

Safetensors

Model size

16B params

Tensor type

BF16

Model tree for Prannesshkva/ISOM-R1-Coder-16B-MoE

Base model

deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct

Finetuned

(19)

this model

Prannesshkva
/

ISOM-R1-Coder-16B-MoE