UmitVice/candela-spark-35m

Candela Spark 35M — milestone 1 (spark-35m-m1)

A 35.0M-parameter decoder trained from scratch in pure MLX on an Apple M5 Pro laptop, quantization-aware from the first step: the int4 weights in this repo are the trained model, not a post-training approximation. Part of Candela, an open financial-analysis LLM project. This is an early base model (no instruction tuning); expect fluent-ish English with weak factual recall.

Files

File	Format
model.safetensors, config.json	MLX affine int4, group 32, bf16 scales/biases — primary artifact
spark-35m-m1-Q4_1.gguf	GGUF Q4_1 (llama.cpp, architecture gemma3) — bit-identical int4 grid
bf16-reference/	bf16 master weights, for reference only
tokenizer.json	frozen byte-level BPE, 32 768 tokens, single-digit splitting
eval.json	full evaluation report

Architecture

Gemma 3 block layout so llama.cpp runs it unchanged: pre- and post-norm RMSNorm around attention and FFN, QK-norm, grouped-query attention (6 query / 2 KV heads, head dim 64), RoPE (base 10k local / 1M global), 5 sliding-window layers (window 256) per global layer, GeGLU FFN, no biases, tied embeddings scaled by √d. d = 384, 12 layers, FFN 1280, context 1024. Parameters: 35,016,576 (12,582,912 in the tied embedding).

Training recipe

• QAT: every matrix (all block linears and the tied embedding/head) is fake-quantized in the forward pass through mx.quantize/mx.dequantize (affine int4, group 32, bf16 scales), the exact grid MLX exports, with a straight-through estimator to fp32 master weights. Active from step 0.
• Optimiser: Muon (Nesterov 0.95, 5 Newton–Schulz steps, decoupled weight decay 0.01, spectral √(fan_out/fan_in) scaling) for hidden matrices; AdamW for the embedding and norms; μP-style output multiplier.
• Schedule: warmup–stable–decay; this release is a 1−√ decay branch forked from the continuous stable run at step 3,600.
• Batch: 32 768 tokens/step (2 × 1 024 × 16 accumulation), bf16 compute, fp32 master weights.
• Trained for 144M tokens (4,400 steps).

Data (pinned, SHA-256 verified, permissive)

Dataset	Revision	License	Files
HuggingFaceTB/smollm-corpus (fineweb-edu-dedup)	3ba9d6057741	odc-by	fineweb-edu-dedup/train-00000-of-00234.parquet
HuggingFaceFW/fineweb-edu (fineweb-edu)	87f09149ef47	odc-by	data/CC-MAIN-2024-51/000_00000.parquet, data/CC-MAIN-2024-51/000_00001.parquet
HuggingFaceTB/smollm-corpus (cosmopedia-v2)	3ba9d6057741	odc-by	cosmopedia-v2/train-00000-of-00104.parquet
wikimedia/wikipedia (wikipedia-en)	b04c8d1ceb2f	cc-by-sa-3.0	20231101.en/train-00000-of-00041.parquet, 20231101.en/train-00001-of-00041.parquet
HuggingFaceTB/finemath (finemath)	e92b25a61673	odc-by	finemath-4plus/train-00000-of-00064.parquet
eloukas/edgar-corpus (edgar)	7e90f0f34256	apache-2.0	2019/train.jsonl, 2020/train.jsonl

Filters: length, English stop-word ratio, alphabetic ratio, word length, repeated lines; EDGAR keeps Items 1, 1A and 7 with page furniture removed; finance/economics pages selected from FineWeb-Edu by a keyword-density rule and upsampled ×3; exact and MinHash (112 permutations, 14 bands) dedup across all sources. Training mix after dedup:

Source	Train tokens	Mixture multiplier
wikipedia-en	310.5M	1.0
edgar	226.8M	2.0
finemath	123.2M	1.0
cosmopedia-v2	255.7M	1.0
fineweb-edu-finance	18.8M	3.0
fineweb-edu-dedup	817.0M	1.0
fineweb-edu	747.3M	1.0

Dataset licenses involved: apache-2.0, cc-by-sa-3.0, odc-by. Wikipedia text is CC-BY-SA-3.0; attribution to Wikipedia contributors.

Evaluation

Validation loss (nats/token) on held-out documents, bf16 master weights vs the int4 QAT forward pass:

Source	bf16	int4	int4 perplexity
cosmopedia-v2	3.2586	3.2617	26.09
edgar	2.3583	2.3619	10.61
finemath	2.9665	2.9694	19.48
fineweb-edu	3.7548	3.7580	42.86
fineweb-edu-dedup	3.5944	3.5937	36.37
fineweb-edu-finance	3.4426	3.4453	31.35
wikipedia-en	3.4882	3.4936	32.9

Mean bf16 3.2662, mean int4 3.2691: relative int4 gap +0.088 %. The exported int4 model reproduces the QAT forward pass (max relative loss difference 0.000279).

Zero-shot (subsets, log-likelihood scoring):

Suite	Metric	n	Score	Chance
lambada	acc	500	0.138	0.00
arc-easy	acc_norm	500	0.352	0.25
hellaswag	acc_norm	500	0.278	0.25
piqa	acc_norm	500	0.582	0.50
blimp	acc	1000	0.869	0.50
finance-cloze	acc	24	1.000	0.25
finance-cloze-hard	acc	16	0.812	0.25

Evaluation sets: EleutherAI/lambada_openai@900124bf3b82 (mit), allenai/ai2_arc@210d026faf99 (cc-by-sa-4.0), Rowan/hellaswag@218ec52e09a7 (mit), ybisk/piqa@142c51238b3c (afl-3.0), nyu-mll/blimp@877fba0801ff (cc-by-4.0). Web-crawl pretraining data may overlap some benchmark text.

Fixed-seed samples (int4, temperature 0.7):

The water cycle describes how a person reacts with water to prevent an injury. It is usually a short walk or a short walk. The water cycle is a short walk or short walk and the water cycle is a short walk or short walk. It is usually done in the form of a stream or stream or stream. It can

In the early nineteenth century, the city of London, where many of the most famous writers and writers were from the English-speaking world, had quite a well-known and unique character. The earlier census, an annual census, was a large and complex affair with the great master of the English language. In the nineteenth century, the Roman Empire lost a significant

Photosynthesis is the process by which light forms, and the process is conducted to produce the desired effects, and, in some cases, the process is performed. The process is a process of photosynthesis and the process is performed by a photosynthesis process. The photosynthesis process is done by a photosynthesis process.

The company reported that revenue for the fiscal year ended May 28, 2018, as a result of the COVID-19 pandemic, is expected to continue to be a new revenue source for fiscal year 2020. The Company’s revenue for fiscal year 2020 was $189 million

When the Federal Reserve raises interest rates, the economy will have to be changed as the Federal Reserve needs to increase its assets. The financial sector is also affected by the federal government’s initiatives, including investments in private and public funds, which are not the primary source of funding. The stock market is affected by the federal government’s decision to increase

Limitations

Tiny base model trained for well under a Chinchilla budget: it makes up facts and numbers, has no instruction following, and must not be used for financial decisions. English only. Knowledge reflects the training snapshots (web crawl to 2024, Wikipedia 2023-11, 10-K filings 2019–2020).

License

Apache-2.0 for the weights and code.