Spark-2 (9.15M Parameters)
Spark-2 is a high-efficiency, sub-10M parameter language model developed by SurjoLabs. With an Intelligence Index of 9.14, it proves that combining 3:1 Grouped-Query Attention (GQA), value-subtraction projection (XSA), and deep recurrent weight-sharing yields exceptional reasoning capabilities at micro scale.
What's New in Spark-2?
- 6:2 Grouped-Query Attention (3:1 GQA): Upgraded from 1 KV head (MQA) to 2 independent Key-Value heads (6 Q heads, 2 KV heads, head dimension 48) to improve contextual memory and multi-step reasoning.
- Expanded Recurrent Depth: Scaled from 3 to 6 recurrent layers (9 physical layers total: 1 prelude + 6 recurrent + 2 coda), delivering an effective computational depth of 15 layers per token across 2 passes.
- Optimal Width & MLP Scaling: Scaled hidden dimension to 288 and intermediate dimension to 768 (2.67x ratio), bringing total parameters to exactly 9,148,608.
- Critical FinePhrase Dataset Fix: In the original Spark v1 pipeline, a configuration bug caused the FinePhrase dataset to map to raw source web data rather than the intended synthetic educational rollouts. In Spark-2, this was resolved alongside strict length filtering (> 24 tokens) to eliminate low-information fragments.
- Performance Leap: Jumped from 7.93 -> 9.14 on the Intelligence Index, notably achieving 37.00% on ArithMark-3.0 and 57.24% on PIQA.
Methodology
Spark-2 demonstrates that specialized recurrent architectures can match or exceed standard transformers multiple times their size when trained on dense, high-quality data.
- Custom Architecture: Built on a Llama-style backbone featuring XSA Attention (orthogonal value-subtraction) and Recurrent Weight-Sharing. By passing activations twice through its 6 recurrent layers, the model achieves the computational depth of a 15-layer model (1 prelude + 6x2 recurrent + 2 coda) while storing only 9 physical layers of weights (~9.15M total parameters).
- Optimized Tokenizer: Uses a custom 4,096-vocabulary English tokenizer. Restricting vocabulary size prevents embedding bloat, keeping ~87% of all parameters dedicated strictly to transformer logic.
- Extreme Overtraining: Trained on ~20.97 Billion tokens (20,000 steps * 1,048,576 tokens/step) to drive dense knowledge compression and syntax mastery.
- Data Mixture:
- 60% FinePhrase (Synthetic FineWeb-Edu, filtered length > 24 tokens)
- 20% DCLM (Curated web text)
- 10% FineMath (Mathematical reasoning)
- 10% CornStack Python (Python code)
- Optimizer: Hybrid setup combining Muon (for 2D weight matrices) with AdamW (for embeddings, lm_head, and normalization layers) with a WSD (Warmup-Stable-Decay) learning rate schedule.
Benchmark Results (Open SLM Leaderboard)
Evaluated at 0-shot using normalized accuracy (acc_norm):
| Benchmark | Spark | Spark-2 |
|---|---|---|
| HellaSwag | 28.17% | 28.13% |
| ARC-Easy | 35.02% | 34.26% |
| ARC-Challenge | 20.99% | 22.18% |
| PIQA | 55.55% | 57.24% |
| ArithMark-3.0 | 36.10% | 37.00% |
| Intelligence Index | 7.93 | 9.14 |
- Evaluations for HellaSwag, ARC-Easy, ARC-Challenge, and PIQA were performed using lm-evaluation-harness (0-shot, acc_norm).
- ArithMark-3.0 evaluated using the official benchmark evaluation script (acc_norm).
Note on Checkpoint
The model was trained for 20,000 steps with a WSD scheduler (decay phase starting at step 16,000). Extensive checkpoint evaluation revealed that checkpoint-17500 achieved peak consolidation across math, commonsense, and logic benchmarks (9.14 INT INDEX). The final step 20,000 was discarded, and step 17,500 is the official version hosted in the root of this repository.
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "SurjoLabs/Spark-2"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
torch_dtype=torch.bfloat16
).cuda()
prompt = "The capital of France is"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=20)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
Acknowledgements
We would like to thank AxiomicLabs for pioneer work proving the token-efficiency of the XSA attention mechanism.
Limitations
Spark-2 is an experimental sub-10M parameter model developed under the broader Surjo Project. While it demonstrates strong reasoning for its size, language generation length and coherence remain constrained by absolute capacity. The modeling files are intended for research and benchmark evaluation.
- Downloads last month
- 340