Spark 2.0-code-base

Spark 2.0-code-base (pretrained base model)

The base model behind Spark 2.0-code: 322M parameters, trained completely from scratch on Python code by one person on a single RTX 3090. It was pretrained on about 18.4B tokens and has no instruction tuning.

This is not a chat model. It does not follow tasks. You give it the beginning of Python code (a function signature with a docstring, an import block, a class header) and it continues it. For instructions in plain English use Spark 2.0-code or Spark 2.0-code (stage 1).

Why release a base model: to fine-tune it yourself, to study it, or to compare it with the instruction-tuned versions.

Honest summary. It completes functions reasonably (HumanEval 34.8%), but it does not know when to stop: after finishing one function it usually goes on and invents more functions. Cut its output yourself (for example at the next top-level def/class), or use a stop condition.

Quick start

Needs an NVIDIA GPU (CUDA). The model uses Gated DeltaNet layers from flash-linear-attention, which run on GPU only.

pip install torch flash-linear-attention triton tokenizers safetensors
python generate.py 'def is_prime(n):\n    # Return True if n is prime.\n'

generate.py prints your prompt and the continuation (greedy, up to 300 tokens). There is no prompt template: the text you give is simply continued.

Model

Parameters 322M (embeddings tied)
Architecture hybrid: Gated DeltaNet layers + full attention every 4th layer, 24 layers, d_model 1024
Context 4096 tokens
Vocabulary 16 384 (own BPE tokenizer, Python-focused)
Pretraining about 18.4B tokens of Python (35 070 steps), Muon + AdamW
Validation loss 0.50
Weights bf16, model.safetensors, 0.65 GB

Results

My own harness, greedy decoding, code is run in a sandbox. HumanEval is completion-style, which is what a base model does.

Base (this model) SFT stage 1 SFT stage 2
HumanEval pass@1 (164 tasks) 34.8% 52.4% 47.6%

HumanEval and MBPP were kept out of the training data: documents sharing a 13-gram with any task or reference solution were removed.

Limitations

  • Not a chat or instruction model; it continues text.
  • Does not stop after a function; trim the output.
  • It sometimes invents library functions and misses imports.
  • It only knows Python.
  • A language model: it can write insecure or wrong code. Review before running.

Training data and license

License: CC BY-NC 4.0 (non-commercial, with attribution). The pretraining data includes KodCode (CC BY-NC 4.0), so I do not offer the model for commercial use. For a commercial product, contact the author first.

Licenses are as shown on the Hugging Face dataset pages when I checked.

Source License
swallow-code-v2 (Python from The Stack v2, rewritten and filtered) Apache-2.0
OpenCodeInstruct (only solutions that passed all unit tests) CC BY 4.0
KodCode-V1 CC BY-NC 4.0
opc-annealing-corpus (synthetic Python snippets) ODC-By

Some of these datasets were themselves generated by other language models, so their own terms may apply. This card is my best effort, not legal advice.

Author

Trained from scratch as a personal project. Feedback and bug reports are welcome.

If you like Spark and want to support the next models (GPU rent for bigger ones), you can donate here: https://boosty.to/sparkcode/donate

Downloads last month
-
Safetensors
Model size
0.3B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support