TassP0lar/Spark-X2.6-4B-Loop-mHC (SparkX-Loop v0.2 Formal Merged Checkpoint)

Self-Contained 60-Effective-Layer Looped Recursive + 3-Tier Memory-Tiered MoE-Engram Transformer built on XHToken/Spark-X2.6-4B (36 physical layers unrolled into 60 virtual layers via Sublayer Dual-mHC and 7 Embedded 3-Tier MoE-Engram Modules = 112 Memory Experts).

• Merged From: TassP0lar/Spark-X2.6-4B-Loop-mHC-R24
• GitHub Repository: BaiTian6641/Spark-X2.6-Recursion
• Physical / Effective Depth: 36 Physical Layers $\rightarrow$ 60 Effective Layers (Prelude [0..4) + 2x Recurrent Core [4..28) + Coda [28..36))
• Learned Gate Telemetry: Mean |attn_gate| = 0.0605 | Mean |ffn_gate| = 0.1087
• 1M Context & Runtime $O(n \log n)$ KV Cache: Preserves 3x sliding_attention (w=512, RoPE) + 1x full_attention (NoPE) with fused GQA SDPA and supports runtime-only $O(n \log n)$ Log-Linear NoPE KV Cache compression (patch_spark_attention_for_loglinear_kv).

Ablation Results on Held-Out Mathematical & Scientific Proofs

Configuration	Effective Layers	Eval Loss	Perplexity	Relative PPL
36-Layer Pure Backbone (Loop OFF, Engrams OFF)	36	1.4648	4.3267	Baseline
60-Layer Loop Only (Loop ON, Engrams OFF)	60	1.4447	4.2406	-2.0%
36-Layer + 7 MoE-Engrams (Loop OFF, Engrams ON)	36	1.4206	4.1395	-4.3%
60-Layer Full (Loop + Dual-mHC + 7 MoE-Engrams)	60	1.4099	4.0957	-5.3% (Best)

Quick-Start Inference (Standalone From This Repo)

import json, torch
from huggingface_hub import snapshot_download
from safetensors.torch import load_file
from transformers import AutoTokenizer, AutoModelForCausalLM
from sparkx_loop_core import LoopSettings, SparkXLoop, patch_spark_attention_for_loglinear_kv

repo_id = "TassP0lar/Spark-X2.6-4B-Loop-mHC"
ckpt_dir = snapshot_download(repo_id=repo_id)
meta = json.loads(open(f"{ckpt_dir}/loop_config.json").read())

tok = AutoTokenizer.from_pretrained(ckpt_dir, trust_remote_code=True)
base = AutoModelForCausalLM.from_pretrained(
    ckpt_dir, trust_remote_code=True, dtype=torch.bfloat16,
    device_map={"": "cuda:0"}, attn_implementation="eager"
)
base.config.use_cache = False

settings = LoopSettings(
    start=meta["loop_start"], end=meta["loop_end"], passes=meta["loop_passes"],
    eng_buckets=meta["engram_buckets"], eng_dim=meta["engram_dim"],
    eng_experts=meta.get("engram_experts", 16), eng_top_k=meta.get("engram_top_k", 2),
    eng_expert_rank=meta.get("engram_expert_rank", 16), eng_shared_rank=meta.get("engram_shared_rank", 16),
    backbone_engram_stride=meta.get("backbone_engram_stride", 4), checkpoint_layers=False
)
net = SparkXLoop(base, settings, patch_sdpa=True).to("cuda:0")
net.load_state_dict(load_file(f"{ckpt_dir}/loop_modules.safetensors", device="cpu"), strict=False)
net.set_loop(meta["loop_start"], meta["loop_end"], meta["loop_passes"], use_mhc=True, use_engram=True)
net.eval()

# Optional: enable runtime-only O(n log n) Log-Linear NoPE KV Cache for ultra-long contexts (>2048 tokens)
# patch_spark_attention_for_loglinear_kv(base)