darioooooo0o/spark-1.7b-engram

💻 DeepSeek Engram (Coding-Specialized) on Spark-X2.5-1.7B

Requests, questions or suggestions? Message me on X: https://x.com/imdariotoo

⚠️ EXPERIMENTAL RESEARCH ARTIFACT: This repository contains an experimental, code-specialized implementation of Conditional Memory via Scalable Lookup (Engram) (arXiv:2601.07372) trained on top of a 100% frozen Spark-X2.5-1.7B model.

---

📥 Direct Downloads & Quick Access

File	Format	Size	Description	Direct Download Link
Engram 25M-Trained Weights	.safetensors	1.6 GB	Full 420.5M-param Coding Engram v2 module	Download Weights (Direct)
Canonical Tokenizer Map	.pt	1.1 MB	Precomputed canonical token mapping	Download Map (Direct)
Training & Eval Metrics	.json	1.7 KB	Step-by-step validation trajectory & PPL	Download Metrics
Full File Tree	—	—	Architecture files & tokenizer assets	Browse All Files & Versions

---

📌 Overview & Multi-Language Scope

This model explores how O(1) conditional memory lookup can augment a frozen compact language model for multi-language code generation and algorithmic reasoning:

• Polyglot Code Memory Bank: The 131k-slot coprime hash tables act as an external memory bank storing multi-token syntax patterns, recursion templates, and standard library idioms across Python, C++, Java, JavaScript, C#, SQL, Bash, and Rust.
• 100% Frozen Backbone (Zero Drift): The base 1.7B transformer weights remain completely frozen. No weights in Spark-1.7B were modified, fine-tuned, or adapted with LoRA.
• Preserved General Reasoning: Because the base model remains untouched, the model experiences zero catastrophic forgetting of its original conversational or non-coding skills.

---

📚 Training Corpus (25,000,000 Multi-Language Tokens)

The Engram module was trained across 25M tokens sourced from diverse programming datasets:

1. nickrosh/Evol-Instruct-Code-80k-v1 (~12M tokens, Multilingual):
   • Complex algorithmic problems, data structures, and competitive programming across Python, C++, Java, JavaScript, C#, Bash, PHP, and SQL.
2. sahil2801/CodeAlpaca-20k + iamtarun/python_code_instructions (~3M tokens):
   • Multi-language task completions, idiomatic one-liners, standard library manipulations, and docstring-to-code implementations.
3. codeparrot/codeparrot-clean (~10M tokens):
   • Clean real-world repository code, modular package architectures, class hierarchies, and production syntax.

---

🏗️ Architecture Specifications

Following DeepSeek's paper, the memory module is structured as follows:

Component	Specification	Description
N-gram Orders	(2, 3, 4)	Multi-scale n-gram context modeling for code tokens
Heads per Order	8 heads (24 heads total per layer)	Multi-head bitwise XOR hashing
Slots per Head	131,072 prime slots	Coprime prime moduli per head eliminating cross-head collisions
Embedding Dimension	64	Compact, high-density idiom memory representation
Target Layers	Layer 2 and Layer 14	Dual-layer topology matching 28-layer Spark-1.7B structure
Tokenizer Compression	131,072 -> 100,096 keys	NFKC + NFD + Accent Strip + Lowercase canonicalization (-23.6%)
Context Gating	Signed Square-Root Gate	sigmoid(sign(S) * sqrt(|S|)) with FP32 RMSNorm
Temporal Convolution	Causal ShortConv (kernel=4, dilation=2)	Strict causal left-padding with SiLU activation and skip connection
Memory Capacity	420.51M Parameters	Sized at ~24.7% of the Spark-1.7B base model

---

📊 Benchmark & Training Results

1. Training & Loss Trajectory (Held-out 1M Validation Shard)

• Step 0 Baseline Val Loss: 1.5396 (Perplexity: 4.66)
• Final Val Loss (Step 1526, 25M tokens): 1.2275 (Perplexity: 3.41)
• Net Improvement: -0.3121 loss reduction (-26.8% perplexity)

2. HumanEval Pass Rate (Greedy, temperature=0.0)

• Stock Spark-X2.5-1.7B Baseline: 7 / 20 (35.0%)
• Spark-1.7B + Native Engram (25M Tokens): 10 / 20 (50.0%, +15.0% absolute gain)

---

🧩 Quantization & Deployment Compatibility

• Weight-Only Quantization (GPTQ / AWQ / INT4 / INT8):
  • Fully Compatible: The frozen base 1.7B transformer weights can be quantized (e.g. GPTQ/AWQ to compress backbone VRAM to ~1.0 GB).
  • Because inter-layer activations (hidden states) remain in 16-bit floating point (bfloat16/float16), the Engram module (enable_engram()) plugs directly into the quantized backbone without numerical degradation.
  • The Engram weights (~1.6 GB safetensors) remain unquantized in bfloat16 / float32.
• GGUF / llama.cpp:
  • Not yet supported in standard llama.cpp: GGUF requires custom C++/GGML tensor graph kernels for Engram's multi-head XOR hashing, prime moduli grid lookup, and Causal ShortConv. Use Hugging Face / PyTorch for Engram inference.

---

🚀 Quickstart & Inference

1. Requirements

pip install torch transformers safetensors

2. Loading the Model with Engram

import torch
from safetensors.torch import load_file
from transformers import AutoTokenizer
# Import architecture definitions from the repository
from configuration_spark import Spark2_5Config
from modeling_spark import Spark2_5ForCausalLM

device = "cuda" if torch.cuda.is_available() else ("xpu" if hasattr(torch, "xpu") and torch.xpu.is_available() else "cpu")
dtype = torch.bfloat16

# 1. Load Base Model
model_id = "XHToken/Spark-X2.5-1.7B"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = Spark2_5ForCausalLM.from_pretrained(model_id, torch_dtype=dtype).to(device)

# 2. Inject Engram Memory at Layers (2, 14)
model.enable_engram(
    target_layers=(2, 14),
    mem_dim=64,
    num_heads=8,
    slots_per_head=131072,
    orders=(2, 3, 4),
    kernel_size=4,
    dilation=2,
    device=device,
    dtype=dtype,
)

# 3. Load Trained Engram Weights
weights = load_file("engram_layer_weights_spark1.7b_25m.safetensors")
for k in weights:
    weights[k] = weights[k].to(device)
model.load_state_dict(weights, strict=False)
model.eval()

# 4. Generate Python Code
prompt = "def greatest_common_divisor(a: int, b: int) -> int:\n    \"\"\" Return a greatest common divisor of two integers a and b \"\"\"\n"
inputs = tokenizer(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=100, temperature=0.0)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

📖 Citation & References

@article{deepseek2026engram,
  title={Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models},
  author={DeepSeek-AI and Peking University},
  journal={arXiv preprint arXiv:2601.07372},
  year={2026}
}