TheCommonAI/common-flash-v0.1

Common Flash v0.1

A drafter for speculative decoding of Ternary Bonsai 8B (1st gen, Q2_0_g64) in llama.cpp. A small model proposes tokens and Bonsai checks them in one pass, so the output is unchanged and decoding is faster.

Qwen3-0.6B-bonsai-lora-Q4_0.gguf (358 MiB) is Qwen3-0.6B, LoRA-tuned on Bonsai 8B's own completions (1,000 completions of at most 192 tokens; rank 32, all linear layers), vocabulary trimmed to Bonsai's 151,669 tokens (llama.cpp needs the draft vocabulary within 128 tokens of the target's), then quantized to Q4_0.

Run it in one command

Needs a recent llama.cpp (llama-server, build 1268eda or newer; see Quick start below). This downloads Bonsai 8B and this drafter, then starts a speculative server with a chat page at http://127.0.0.1:8080:

llama-server \
  -hf  prism-ml/Ternary-Bonsai-8B-gguf:Q2_0_g64 \
  -hfd TheCommonAI/common-flash-v0.1 \
  -ngl 99 -ngld 99 -fa on -np 1 -c 4096 \
  --spec-type draft-simple --spec-draft-n-max 3

-hf fetches the target model, -hfd fetches the drafter from this repo. Tested from an empty cache; the first start downloads about 2.7 GB.

About the "Use this model" button. Hugging Face generates that snippet itself, and for this repo it would start the drafter alone, which is not useful. Use the command above instead.

Target model (required pairing)

This drafter only works with Ternary-Bonsai-8B-Q2_0_g64.gguf from prism-ml/Ternary-Bonsai-8B-gguf (2,310,125,920 bytes, sha256 starting e17b298d84ee). That is first-generation Ternary Bonsai 8B. It was tuned on that model's outputs and trimmed to its vocabulary; it is not tested with other Bonsai models (Bonsai 2, the 27B, or the 1.7B), and mismatched vocabularies are rejected by llama.cpp.

Quick start

You need three things: a recent llama.cpp, the Bonsai 8B model, and this drafter.

1. Build or install llama.cpp (recent upstream). Tested with upstream commit 1268eda. An older Homebrew build (b11429) was slower with speculation than without, so use a current build.

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build            # Apple Silicon: Metal is on by default
# NVIDIA GPU:  cmake -B build -DGGML_CUDA=ON      (Vulkan: -DGGML_VULKAN=ON)
cmake --build build --config Release -j
# binaries land in build/bin/ (build/bin/Release/ on Windows)

2. Download both models.

pip install -U "huggingface_hub[cli]"
hf download prism-ml/Ternary-Bonsai-8B-gguf Ternary-Bonsai-8B-Q2_0_g64.gguf --local-dir models
hf download TheCommonAI/common-flash-v0.1 Qwen3-0.6B-bonsai-lora-Q4_0.gguf --local-dir models

Disk: about 2.3 GB for Bonsai plus 0.4 GB for the drafter. Memory: roughly 3 GB plus the context cache.

3. Run the server.

./build/bin/llama-server \
  -m  models/Ternary-Bonsai-8B-Q2_0_g64.gguf \
  -md models/Qwen3-0.6B-bonsai-lora-Q4_0.gguf \
  -ngl 99 -ngld 99 -fa on -t 6 -np 1 -c 4096 \
  --spec-type draft-simple --spec-draft-n-max 3 \
  --host 127.0.0.1 --port 8080

Flag	Meaning
-m / -md	Target model (Bonsai) / draft model (this repo)
-ngl 99 / -ngld 99	Put all layers of the target / drafter on the GPU. Use 0 for CPU-only (untested, likely slower)
-fa on	Flash attention
-t 6	CPU threads; set to your number of performance cores
-np 1	One request slot. Speculation helps single-stream speed; more slots share the GPU
-c 4096	Context size in tokens. Lower it (for example -c 2048) if you run out of memory
--spec-type draft-simple	Classic draft-model speculative decoding
--spec-draft-n-max 3	Draft 3 tokens per round. 3 was best in our tests; 8 was close; 2 and 5 were no better

4. Try it. Open http://127.0.0.1:8080 for the built-in chat page, or:

curl -s localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "messages": [{"role": "user", "content": "Explain speculative decoding in two sentences."}],
  "max_tokens": 128, "temperature": 0.7, "top_p": 0.8, "top_k": 20, "presence_penalty": 1.5,
  "chat_template_kwargs": {"enable_thinking": false}
}'

5. Check that speculation is working. The server's own /completion endpoint reports it:

curl -s localhost:8080/completion -d '{"prompt":"<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n","n_predict":64}' \
  | python3 -c "import json,sys; t=json.load(sys.stdin)['timings']; print(t['predicted_per_second'],'tok/s', t.get('draft_n_accepted'),'/',t.get('draft_n'),'drafts accepted')"

Acceptance of roughly 60-70% on code and copy-heavy text, and 35-45% on open chat, is normal. If draft_n is missing or 0, the drafter did not load.

To compare against plain decoding, start a second server on another port without -md, -ngld and --spec-type, and send both the same prompts. Run them one after the other on a quiet machine: absolute speeds on a busy machine are noisy, and only ratios are meaningful. Speculation cannot be switched off per request in this build, so restart the server to disable it.

Troubleshooting

• "draft model vocab must match" / vocab size error: you are not using the exact Bonsai 8B 1st-generation Q2_0_g64 file named above.
• Slower than without the drafter: use a current upstream llama.cpp build; try --spec-draft-n-max 2; make sure nothing else is using the GPU.
• Out of memory: lower -c, and run one server at a time.
• Output differs from plain decoding: with temperature 0 the text is identical. With sampling the random draws differ, which is expected.

Platform status

Platform	Status
Apple Silicon (Metal)	Measured, about 1.36x (one M3, 16 GB)
NVIDIA CUDA (Linux/Windows)	Bonsai Q2_0 runs on CUDA; speculation speedup not yet measured
Vulkan, CPU-only	Untested

If you try another platform, please report your plain versus speculative tokens per second.

Results

One Apple M3 (16 GB), Metal, 36 prompts the drafter never trained on, 128 tokens each, the network worker's sampling (temperature 0.7, top_p 0.8, top_k 20, presence_penalty 1.5), configurations interleaved prompt by prompt. Speed relative to plain decoding:

	speedup
overall	~1.36x
chat	~1.14x
code	~1.31x
copy-heavy edits	~1.67x

Draft acceptance is about 65% overall but only 33-45% on chat. Run-to-run ratios varied between 1.19x and 1.41x on a busy machine, so treat these as approximate. With greedy decoding every speculative output matched plain decoding. With sampling the random draws differ, so that check does not apply.

Charts

Decode speed on the same M3 (plain, llama.cpp 1268eda, llama-bench tg128)

Model	File size	tokens/s
Ternary Bonsai 1.7B	0.46 GB	127
Ternary Bonsai 2 2B	0.51 GB	97
Common Flash (Bonsai 8B + drafter)	2.3 GB + 0.4 GB	~49 (36.3 x 1.36)
Ternary Bonsai 2 4B	1.1 GB	48
Ternary Bonsai 8B, plain	2.1 GB	36
Ternary Bonsai 2 9B	2.4 GB	28

The Common Flash row is derived: plain Bonsai 8B speed times the measured 1.36x ratio, not a separate llama-bench run. Smaller models are faster but are different, smaller models; the comparison that matters for quality is Bonsai 8B plain against Bonsai 8B with the drafter, which produce the same text.

Limits

• Tested on one machine. Older llama.cpp builds or other hardware may not see a speedup.
• The test prompts come from Alpaca-cleaned, CodeAlpaca and HumanEval-style prompts, not real traffic.
• More training data, soft-label distillation, tighter quantization, MLP pruning and tuned draft lengths did not improve it; acceptance appears limited by the 0.6B drafter's size.

Licence

Apache-2.0, inherited from Qwen3-0.6B. Training inputs were prompts from Alpaca-cleaned and CodeAlpaca-20k with Bonsai 8B's own completions as targets; check those datasets' terms and Prism ML's Bonsai terms before relying on this commercially.