
Spark 2.0-code
A small language model (322M parameters) that writes Python code only. It was trained completely from scratch (pretraining, then two SFT stages) by one person on a single RTX 3090.
You describe a task in English, it answers with Python code.
Three versions. Spark 2.0-code (this one) writes complete programs best. Spark 2.0-code (stage 1) has the highest HumanEval (52.4%) and is best for single functions. Spark 2.0-code-base is the pretrained base model that only continues code.
Honest summary. Spark is good at functions, short scripts and console programs with a menu or a loop. It is weak at long programs where many parts must fit together (games like Snake, big multi-class apps). It often writes correct-looking code that fails on one detail. Always run and check what it writes.
Quick start
Needs an NVIDIA GPU (CUDA). The model uses Gated DeltaNet layers from flash-linear-attention, which run on GPU only.
pip install torch flash-linear-attention triton tokenizers safetensors
python generate.py "Write a Python function that checks whether a number is prime."
generate.py loads model.safetensors and uses the code in spark/. Generation is greedy (temperature 0).
Prompt format (the same as in training):
### User
<task in English>
### Assistant
The answer is plain code, followed by <|endoftext|>. Multi-turn works in the same format: after the answer add a new
### User block, for example with an error message ("The code above fails with this error: ... Fix the code.").
Ask "What model are you?" and it answers that it is Spark 2.0-code.
Model
| Parameters | 322M (embeddings tied) |
| Architecture | hybrid: Gated DeltaNet layers + full attention every 4th layer, 24 layers, d_model 1024 |
| Context | 4096 tokens |
| Vocabulary | 16 384 (own BPE tokenizer, Python-focused) |
| Pretraining | about 18.4B tokens of Python, Muon + AdamW |
| SFT | stage 1 (functions) and stage 2 (complete programs, fixing code from an error message) |
| Weights | bf16, model.safetensors, 0.65 GB |
Results
All numbers come from my own harness in this project, not from an external leaderboard. Greedy decoding, code is run in a sandbox.
| Pretrained base | SFT stage 1 | SFT stage 2 (this model) | |
|---|---|---|---|
| HumanEval pass@1 (164 tasks) | 34.8% | 52.4% | 47.6% |
HumanEval tests only single functions. Stage 2 moved the model towards complete programs, and its HumanEval score is about 5 points lower than stage 1 (8 tasks out of 164; part of that can be noise).
My own small checks of the same model (few attempts each, so treat them as rough):
| Check | Stage 1 | Stage 2 |
|---|---|---|
| Console calculator with all 5 operations working (20 tries) | 1/20 | 12/20 |
| "Guess the number" game works (10 tries) | 10/10 | 8/10 |
| 10 short function tasks, greedy | 10/10 | 10/10 |
| Snake in pygame survives 300 frames (8 tries) | 0/8 | 1/8 |
HumanEval and MBPP were kept out of the training data: documents sharing a 13-gram with any task or reference solution were removed.
Limitations
- Long programs (100-200 lines) usually contain several small bugs: a wrong argument count, a string used where an object is needed, a forgotten import, a method that is never called. Examples I tested: a library manager, a school journal, a pygame arcade game. The structure was fine, but most menu commands crashed on first run.
- It cannot reliably do geometry (positions and sizes of shapes) or game logic where many things depend on each other.
- It sometimes invents library functions and misses imports.
- It only knows Python. English prompts work best.
- A language model: it can write insecure or wrong code. Review before running.
Training data and license
License: CC BY-NC 4.0 (non-commercial, with attribution). Reason: the model was trained on data that includes KodCode (CC BY-NC 4.0), so I do not offer it for commercial use. If you want to use Spark in a commercial product, contact the author first.
Data used (all text only Python-related; licenses as shown on their Hugging Face pages when I checked):
| Stage | Source | License |
|---|---|---|
| Pretraining | swallow-code-v2 (Python from The Stack v2, rewritten and filtered) | Apache-2.0 |
| Pretraining | OpenCodeInstruct (only solutions that passed all unit tests) | CC BY 4.0 |
| Pretraining | KodCode-V1 | CC BY-NC 4.0 |
| Pretraining | opc-annealing-corpus (synthetic Python snippets) | ODC-By |
| SFT | OpenCodeInstruct, KodCode-V1, OpenCoder opc-sft-stage1/2, Magicoder OSS-Instruct and Evol-Instruct, Code-Feedback, CodeFeedback-Filtered-Instruction, glaive-code-assistant-v3, self-oss-instruct-sc2-exec-filter-50k, LeetCodeDataset | MIT, Apache-2.0, CC BY 4.0, ODC-By, CC BY-NC 4.0 (KodCode) |
| SFT stage 2 | code_contests (Python 3 solutions, checked by running the tests) | CC BY 4.0 |
| SFT stage 2 | about 3 800 complete programs written by Qwen3.8-Flash-Next (local, quantized) and checked by running them in a sandbox | Qwen model license not verified by me |
| SFT stage 2 | about 1 200 "fix it from the error message" dialogues made from the real mistakes of stage 1 of this model | own |
Some of these datasets were themselves generated by other language models, so their own terms may apply. This card is my best effort, not legal advice.
Author
Trained from scratch as a personal project. Feedback and bug reports are welcome.
If you like Spark and want to support the next models (GPU rent for bigger ones), you can donate here: https://boosty.to/sparkcode/donate
- Downloads last month
- 30