Parakon/parakon-runtime

Parakon Runtime

The inference runtime for Parakon compressed models: a fork of llama.cpp with custom CUDA kernels for Parakon's 1-bit storage formats. Stock llama.cpp cannot load these artifacts; this runtime runs them at full speed with the weights kept compressed in GPU memory end to end — they are never expanded to fp16.

What it runs

Model	Artifact	Hardware measured	Decode speed
Parakon-122B	23.9 GB	4× NVIDIA A10G (25.0 GB VRAM total)	64.5 tok/s
Parakon-30B	5.9 GB	1× NVIDIA A10G (6.5 GB VRAM)	162.9 tok/s

All figures measured with this repository at 4096 context, greedy decoding. Metal (Apple silicon) and CPU execution of Parakon types are not in this repository yet; this release is CUDA. A CUDA-capable GPU is required. The 122B fits and runs on a single g5.12xlarge; the 30B fits on any 8 GB GPU.

Build (CUDA)

cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j

Run

./build/bin/llama-completion -m parakon-30b.gguf -ngl 99 -c 4096 \
  -p "Explain why the sky is blue."

Multi-GPU works out of the box; layers are split automatically. Sampling recommendations are on each model card.

How it works

Parakon models store MoE expert weights in 1-bit codebook formats. The decode path uses fused mat-vec kernels that read the quantized blocks directly from VRAM and accumulate against int8-quantized activations via dp4a — no dequantized copy of the weights ever exists in memory. Prompt processing at large batch sizes falls back to a dequant-per-tile GEMM path automatically.

Attribution

• Fork of ggml-org/llama.cpp (MIT — see LICENSE). Upstream README preserved as README_llama.cpp.md; all upstream copyright notices retained.
• The 1-bit block formats follow the design of Unsloth's dynamic 1-bit quantization family. The kernels, the fused decode path, and the compression pipeline that produces Parakon artifacts are Parakon's own work.

License

The llama.cpp base remains MIT (LICENSE). Parakon's additions (files prefixed parakon, and the Parakon dispatch changes) are released under the Parakon Community License (LICENSE_PARAKON): free for research, evaluation and personal use, free commercial use for organizations under 100 employees and $1M annual revenue; larger organizations need a commercial agreement — contact the Parakon team via the org page.