Grimlabs/GrimBoost-01

GrimBoost-01 is the first release of the GrimBoost drafter: a 19.1M-parameter, target-agnostic model that proposes output bytes for a local LLM, together with the runtime that decides when those proposals can skip the target and how to repair them when they cannot.

Most speculative decoders verify every drafted token. GrimBoost is built around a different question: how much target work can be eliminated, and what does the resulting drift cost? High-confidence spans can be released without a target call. A cascade — invalidation, doubt intervals, mandatory repair verification, and bounded takeover — contains the errors that follow. The output quality trade is measured as slip, not assumed away.

Measured today: 1.40×–2.90× end-to-end throughput versus the same target in eager Transformers, with 100% emitted-byte agreement in every reported run. Those gains come from target execution optimizations. The drafter has not qualified for unchecked release, so zero unverified draft tokens were emitted.

At a glance

Parameters	19,126,656 · FP32 · 76.5 MB
Architecture	8-layer decoder · GQA 6 query / 3 KV heads · RoPE θ=100,000 · SwiGLU · RMSNorm
Tokenizer	Own 16,000-token byte-level BPE — no shared vocabulary with the target
Alignment	UTF-8 bytes · immutable target token path · target-first-token rule
Target families	Local HF causal LMs: Qwen3, Qwen3.5, Llama, Mistral, GPT-2 / GPT-NeoX
Measured speed	1.40×–2.90× vs eager Transformers on 10k-token workloads
Output agreement	100% of emitted bytes in the reported runs
Prediction status	Not certified for unchecked generation
License	Apache-2.0 · © 2026 Grimlabs LLC

How it works

1. Draft. The 19M model drafts a block of bytes and records per-token confidence, top-1 probability, margin, and entropy.
2. Gate. Confidence is aggregated per target-token span using the minimum overlapping drafter confidence. Anchors, doubt intervals, unverified-byte caps, and sampling mode can force verification regardless of confidence.
3. Verify. When a span is gated, the target processes the contiguous span since its last synced frontier in one batched pass. UTF-8 bytes are canonical; the target's token path is never re-tokenized. Only the first mismatch is actionable.
4. Cascade. A correction commits the target's full token, invalidates every downstream candidate, opens a doubt interval, re-drafts the suffix, and requires target verification of the repair span. Progress is monotone; repeated failure triggers a bounded target takeover.
5. Reveal. Strict mode releases only the maximal verified prefix. Streaming mode releases confident spans immediately, leaves the target cache behind, and audits released bytes later. Escaped bytes cannot be retracted.

Mode	Target work	Output	Intended use
Strict	Every emitted token is verified	Byte-exact greedy control	Correctness, parity tests, conservative experiments
Streaming	Confident spans skip the target	Approximate; escaped slip is possible and recorded	Target-call-elimination research

Measured performance

All arms use the same target weights and quantization: RTX 4070 12 GB, NF4 double quantization, BF16 compute, greedy decoding, FP32 drafter. Each workload contains 10,000 fresh native target tokens. Times include prefill, runtime setup, and online learning/qualification; model and checkpoint loading are excluded.

Local target	Eager	Optimized + learning	Gain	Emitted-byte agreement
Qwen3-0.6B	112.718 s	38.840 s	2.902×	31,998 / 31,998
Qwen3.5-2B (hybrid)	168.346 s	82.268 s	2.046×	34,856 / 34,856
Qwen3-4B-Instruct-2507	166.605 s	118.632 s	1.404×	40,604 / 40,604

Learning and qualification add 6.009 s on the 0.6B workload (0.601 ms per native token) and 10.116 s on the 2B workload (1.012 ms per native token), measured against the optimized target alone.

Where the speed comes from. These gains come from target execution — CUDA graphs, packed compatible NF4 Q/K/V and gate/up projections without requantization, fused KV-write/RMSNorm kernels, and a validated hybrid recurrent path. They are not a demonstration of unchecked speculative generation. The readiness gate failed in every reported learning arm, so the target produced every emitted byte.

These comparisons are against eager Transformers, not vLLM, llama.cpp, TensorRT-LLM, or another optimized serving engine. They are complete 10k-token workload measurements, not isolated post-warmup timing.

Current frontier

The drafter itself is not yet a speed win. With this 19M checkpoint, quality-preserving operating points measured 0.51–0.75× target-only throughput. Draft-only generation reached 6–8× at short context and 2.7–4.3× at long context, but with 0.32–0.58 token slip and near-zero byte match. Closing that gap — rather than tuning the benchmark — is the next research step.

Prediction status

The 19M drafter is included as a research artifact. It is not certified for unchecked generation, and loading it alone does not accelerate another model.

Target / experiment	Student-token agreement	Confident coverage	Agreement at coverage	Readiness
Qwen3-0.6B catalogue validation	87.743%	24.313% at p ≥ 0.999	99.835% (2 errors / 1,212)	Failed — Wilson 95% upper error bound 0.600%, required ≤ 0.100%
Qwen3-4B structured JSON	≈21.7%	No qualifying coverage	—	Failed
Qwen3.5-2B hybrid candidate	—	—	—	Failed

The 100k-token observation-budget experiment also failed its combined readiness gates. These are student-token metrics after byte alignment; they are not native-target token-identity scores. Results depend strongly on domain, target, context, and training budget.

What “100% agreement” means

Every output byte in the measured optimized runs matched its paired eager reference. This measures output agreement on those workloads. It is not a factual-accuracy score, a proof for every prompt, or 100% prediction accuracy by the drafter. Floating-point backend changes can alter close greedy decisions; check parity on your model, device, precision, and workload.

Quickstart

Download or clone this repository and run from its root. It includes the runtime source, so there is no dependency on a similarly named PyPI package. Use Python ≥3.12 with a working CUDA PyTorch installation. The exact measured environment is in requirements-tested.txt.

python -m venv .venv && source .venv/bin/activate
pip install -e .

# Target weights download from the Hub if not already cached.
grimboost generate \
  --target Qwen/Qwen3-0.6B --quantize 4bit \
  --drafter-checkpoint . \
  --target-runtime auto --decode-optimizations auto \
  --chat-template native --mode strict \
  --prompt "Explain binary search in a short paragraph." \
  --max-new-bytes 1024

Swap --target for Qwen/Qwen3.5-2B or Qwen/Qwen3-4B-Instruct-2507 when memory permits. --target-runtime auto selects a supported CUDA backend; eager reproduces the original target runtime. --decode-optimizations auto enables the v2 kernels for output budgets ≥8,192 bytes; set on to force them where supported. If your Python lacks ensurepip, the tested fallback is uv venv --system-site-packages .venv && source .venv/bin/activate.

To load the published drafter directly:

grimboost generate \
  --target Qwen/Qwen3-0.6B --quantize 4bit \
  --drafter-checkpoint Grimlabs/GrimBoost-01 \
  --drafter-revision <commit-sha> \
  --chat-template native --mode strict \
  --prompt "Write a haiku about gradients." --max-new-bytes 512

Pin --drafter-revision to an immutable commit for reproducible runs.

Inspect the drafter with Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "Grimlabs/GrimBoost-01"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=False)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=False, attn_implementation="sdpa"
).eval()

inputs = tokenizer("The next item is", return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=24, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))

The export maps to standard LlamaForCausalLM modules with no custom remote code. model_type: llama describes the architecture; GrimBoost was trained from scratch and is not a Meta Llama derivative. FP32 full-forward, cached-step logits, and greedy generation are checked against the native drafter. Reduced-precision HF execution is not parity-certified because normalization arithmetic can differ.

AutoModelForCausalLM.generate runs the drafter alone. It does not enable confidence gating, target verification, or acceleration — use the GrimBoost runtime for that.

Architecture and training

Property	Value
Parameters	19,126,656 (tied embeddings counted once)
FP32 storage	76,506,624 bytes ≈ 76.5 MB
Layers / hidden / MLP	8 / 384 / 1,024
Query heads / KV heads / head dim	6 / 3 / 64
Attention / activation / norm	Causal GQA / SwiGLU / RMSNorm, ε = 1e-6
Position encoding	Half-rotation RoPE, θ = 100,000
Tokenizer	Own byte-level BPE, 16,000 tokens
Special IDs	EOS 0 · PAD 1 · BOS 2
Configured context	32,768 positions
Pretraining sequence length	4,096 tokens (long-context fine-tuning not run)
Checkpoint	Step 4,000 · ≈1.048B token exposures
Validation loss	2.028 on the project mixture

The packed corpus contains ≈1.353B unique tokens. The sampling mixture was UltraChat SFT 40%, FineWeb-Edu 30%, CodeParrot-clean 20%, and MetaMathQA 10%. Training was trained from scratch with no target-generated labels, no logit distillation, and no per-target gradients. It stopped at the 1B-token go/no-go, before the planned 3.015B exposures and the long-context stages. These datasets can contain errors and biases; the drafter has no independent factual or safety evaluation. See the training report and the dataset cards linked in the Hub metadata.

Weights use safetensors. Optimizer states, RNG states, API keys, raw training data, and other local target weights are excluded. release_manifest.json records hashes; grimboost_metadata.json records checkpoint/tokenizer provenance; validation.json contains checks against the actual exported weights.

Compatibility

Target family	Execution / correction support	Evidence
Qwen3 full attention	CUDA graphs, supported NF4 fusion, native cache rollback	Pretrained 0.6B and 4B measurements
Qwen3.5 hybrid text	Native recurrent CUDA capture, checkpoint + replay correction	Pretrained 2B measurements; no vision workload tested
Llama full attention	CUDA SDPA capture and rollback	Synthetic models; no pretrained speed claim
Mistral, sliding disabled	CUDA SDPA capture and rollback	Synthetic models
Mistral sliding / Llama dynamic RoPE	Native fallback with guarded replay	Synthetic models
GPT-2 / GPT-NeoX	Native fallback and cache restoration	Synthetic models
Other HF causal decoders	Depends on model, tokenizer, and cache interfaces	Evaluate individually

Tokenizer support includes the GPT-style byte alphabet used by the tested Qwen models and a guarded SentencePiece byte-fallback profile. Other profiles fail explicitly. Byte alignment does not make every architecture, tokenizer, quantization, or hardware backend automatically supported.

The measured accelerator requires local NVIDIA CUDA execution. CPU can load the exported drafter; CPU, Apple Metal, AMD, multi-GPU serving, vision, and arbitrary quantization speed gains have not been established. Memory must fit the target, its cache, graph state, drafter, and optimizer; 76.5 MB is only the drafter's FP32 weights.

What this is — and is not

✅ This is	❌ This is not
A 19M target-agnostic drafter plus a local runtime for target-call elimination research	A drop-in accelerator for hosted APIs or arbitrary models
Byte-exact in strict greedy mode, with target-first-token alignment	A lossless guarantee for streaming or sampling mode
Measured on Qwen3-0.6B, Qwen3.5-2B, and Qwen3-4B	Proof of model-agnosticism for every architecture
A measured speed/agreement trade with published negative results	A universal ×50 speedup claim

Honest limits

• Strict verification is a correctness control, not a speed path. The measured gains above come from optimized target execution; strict sequential speculation has not beaten target-only throughput in these experiments.
• Streaming is approximate. Anchors bound detection delay, not error probability. Detected errors can be repaired; escaped bytes cannot be retracted.
• Batched verification is experimental. Numerical differences can change greedy decisions; the default verifier is sequential.
• Sampling is tolerance-based. Cross-tokenizer speculative sampling is not exact.
• The target is a local 4B model, not a frontier model. This demonstrates the mechanism on open weights; larger targets require more memory and were not measured.
• One target family is not model-agnosticism. The architecture and training are target-independent, and the engine fails fast on unsupported caches, but broad compatibility remains an empirical question.
• Hosted APIs do not benefit. The tested fixed endpoint rejected external prediction verification; no same-provider 1.5× API gain is demonstrated.

Reproduce

# Reconstruct the native checkpoint layout from this HF release (no optimizer state).
python scripts/restore_native_checkpoint.py --release . \
  --out outputs/pretrain_19m_4k/last.pt \
  --tokenizer-out data/tokenizer/bpe16k.json

# HF-format interoperability only (CPU).
python -m pytest tests/test_huggingface.py

# Full local benchmarks (GPU; run one at a time).
python scripts/benchmark_learning_transfer.py --model Qwen/Qwen3-0.6B \
  --tokens 10000 --out outputs/local_06b.json
python scripts/benchmark_verified_v2.py --repeats 2
python scripts/benchmark_learning_transfer.py --model Qwen/Qwen3.5-2B \
  --tokens 10000 --out outputs/model_transfer/qwen35_2b_learning.json
python scripts/benchmark_hybrid_speed.py

Compare the same model revision, precision, prompt bytes, EOS behavior, output budget, and execution conditions. Report loading/setup inclusion, repeat count, elapsed time, full byte equality, readiness, unchecked output count, and target work.

Raw timing JSON and historical source snapshots are under benchmarks/. Design notes and full experiment reports are under docs/.

Files

Path	Contents
LICENSE / NOTICE	Apache-2.0 terms and Grimlabs LLC attribution
model.safetensors	FP32 drafter weights
config.json	Llama-compatible architecture config
tokenizer.json / tokenizer_config.json	16k byte-level BPE
chat_template.jinja	Canonical ChatML-style prompt template
grimboost_metadata.json	Provenance, hashes, observation budget
validation.json	Export validation against native weights
release_manifest.json	SHA-256 for every release file
docs/	Design, experiments, reports, limits
scripts/	Reproduction and benchmark scripts
benchmarks/	Raw timing evidence and source snapshots
requirements-tested.txt	Exact measured environment

Citation

@misc{grimboost01,
  title        = {GrimBoost-01: Confidence-Gated Selective Verification with
                  Cascading Re-Evaluation},
  author       = {{Grimlabs}},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/Grimlabs/GrimBoost-01}},
  note         = {19.1M-parameter target-agnostic drafter; prediction not certified}
}

License

Copyright 2026 Grimlabs LLC.

GrimBoost-01 — including the model weights, runtime, and release materials — is licensed under the Apache License, Version 2.0. See LICENSE and NOTICE.

The license does not grant rights to third-party training datasets or target models; their upstream terms still apply. “GrimBoost” and “Grimlabs” are trademarks of Grimlabs LLC; the Apache License does not grant trademark rights beyond customary use in describing the origin of the work.