opencoti-llamafile

Self-contained single-file inference engine from the opencoti project — a Mozilla-Ocho llamafile 0.10.3 base carrying the opencoti patch series: advanced KV residency/quantization (PolyKV/TurboQuant/TCQ), rolling-KV window with host-RAM spill, DCA long-context extension, MTP speculative decode, sparse attention, RYS layer duplication, and a runtime introspection/control API.

Start with USAGE.md — it explains exactly how this engine diverges from upstream llamafile, every added feature, its flags, defaults, limitations, and which features compose.

This repo hosts the packaged release artifacts (they exceed GitHub's 2 GiB release-asset cap) plus the full patch series under patches/ and the project documentation under docs/. Release notes and per-release SHA256SUMS live under releases/ and are mirrored on the corresponding GitHub release at mann1x/opencoti.

Supported / target model families

This engine is not model-agnostic in what it optimizes. Upstream llama.cpp GGUF support is unchanged (any GGUF that loads upstream loads here), but the opencoti feature set is built, tuned, and validated on two families:

Family	Role	Models	What's tuned for them
Gemma-4	PRIMARY target	26B-A4B-128e MoE ("A4B"), dense 12B / 31B, elastic E-series (E2B/E4B)	head_dim 256/512 kernels, iSWA dual-cache (rolling-KV, SharedKVPool, DCA wiring), MTP via gemma4-assistant drafters, fused MoE
Qwen	SECONDARY / verification	Qwen3.5 / 3.6 dense + MoE + hybrid (gated-delta-net), Qwen2.5-14B-1M	head_dim 128 kernels, NextN self-spec MTP (fused multi-step draft), DCA long-context on 1M-class models

Other architectures run with upstream behavior; opencoti features either fall back safely or are unvalidated on them — see the model-families section at the top of USAGE.md before relying on an opencoti feature elsewhere.

Artifacts

Artifact	Platform	GPU
opencoti-llamafile-<ver>-<tag>-x86_64.llamafile	x86_64 + aarch64 (fat APE; Linux/macOS/BSD)	CUDA 13.3 embedded, x86_64-linux (sm_75/80/86/89/90/120f)
opencoti-llamafile-<ver>-<tag>-win-x86_64.llamafile.exe	same fat APE, incl. Windows	none embedded — CPU out of the box, GPU via side-load (below)
opencoti-llamafile-<ver>-<tag>-aarch64.llamafile	same fat APE	CUDA 13.3 embedded, sbsa/aarch64-linux (sm_110f DGX Spark GB10 / Jetson Thor, sm_121a)
dso/<ver>-<tag>/ggml-cuda-x86_64.so	side-load DSO for the -win/bare APE on x86_64-linux	CUDA 13.3, x86_64-linux (sm_75/80/86/89/90/120f)
dso/<ver>-<tag>/ggml-cuda-sbsa-aarch64.so	side-load DSO for the -win/bare APE on aarch64-linux	CUDA 13.3, sbsa/aarch64-linux (sm_110f, sm_121a)

The APE host binary is identical across variants and runs natively on both x86_64 and aarch64. The variants differ ONLY in the embedded ggml-cuda.so: the full x86_64 artifact carries the x86_64-linux DSO (4.8 GB); the -aarch64 artifact carries the sbsa/aarch64-linux DSO (1.8 GB — GPU out of the box on DGX Spark-class hosts); the -win variant carries none, because Windows refuses to run executables larger than 4 GB — it is the same APE renamed .exe (required on Windows) at ~55 MB. With no matching DSO available, inference falls back to CPU.

Release layout: only the latest cut's artifacts live at the repo root (and its DSOs under dso/<ver>-<tag>/). Older cuts are moved whole — artifacts, MANIFESTs, DSOs, notes, SHA256SUMS — to releases/archived/, one folder per release. Release notes for the current cut are under releases/.

Usage

chmod +x opencoti-llamafile-*.llamafile
# server mode
sh ./opencoti-llamafile-*.llamafile --server --port 8080 \
    -m <model>.gguf -ngl 99 --flash-attn on
# without --server it starts the interactive chat CLI

On the first GPU run the embedded CUDA DSO self-extracts to ~/.llamafile/v/<ver>/. --version prints the opencoti version string; GET /props exposes an opencoti introspection section at runtime (see USAGE.md §introspection).

-win variant / GPU side-load

• Windows: run opencoti-llamafile-<ver>-<tag>-win-x86_64.llamafile.exe directly (CPU inference works out of the box). For GPU, install the NVIDIA CUDA Toolkit + MSVC; on first -ngl run llamafile compiles a native ggml-cuda.dll on the fly into %USERPROFILE%\.llamafile\.

• Linux x86_64 (bare APE + GPU without the 4.8 GB download’s re-download): drop the published side-load DSO where the engine looks before extracting an embedded one:

  mkdir -p ~/.llamafile/v/<ver>/   # e.g. 0.10.3
  cp ggml-cuda-x86_64.so ~/.llamafile/v/<ver>/ggml-cuda.so
  

  Verify against releases/<tag>/SHA256SUMS first. The -win file is still a normal APE — sh ./…​.exe runs fine on Linux/macOS/BSD.

Verification

Each artifact ships with a MANIFEST.json recording the artifact sha256, versionString, gitCommit, the full patch list, and the sha256 of every embedded backend DSO; releases/<tag>/SHA256SUMS is the committed contract (CI verifies this repo's LFS blobs against it on every release tag). Verify the embedded DSO without running:

unzip -p opencoti-llamafile-*.llamafile ggml-cuda.so | sha256sum

License

Apache-2.0 (llamafile) + MIT (llama.cpp and bundled projects). opencoti patches and packaging are MIT.

PolyKV multi-session stress benchmark (package_courier)

An agentic multi-session benchmark: LangGraph courier agents (tool-calling, 10-step delivery chains, 100 packages) run against ONE engine at --parallel 32, ctx 163840, q8_0 KV, with a tps-floor scheduler that admits new sessions only while every live session keeps ≥ floor tok/s (PolyKV /polykv/tps SSE telemetry drives admission). Each worker episode is a compiled langgraph.StateGraph (capacity_gate → agent ⇄ tools, with the capacity-gate node reading the engine's /polykv/capacity), so the harness doubles as the reference LangGraph integration for this engine. The full harness is published in this repo at bench/package_courier/ (work server + LangGraph agent app + launcher + HTML report generator — see its README and the usage guide at docs/features/package_courier_harness.md); reports under bench/reports/ (per-run HTML + raw stats).

The opencoti_langgraph Python package the harness builds on — OpencotiChatModel (LangChain BaseChatModel with PolyKV pool/session attach), PolykvClient/AsyncPolykvClient, pool tools, and the capacity-gate / compaction LangGraph nodes — is published at integrations/langgraph/ (pip install ./integrations/langgraph, needs langchain-core>=1.4.9

• langgraph>=1.2.9; see its README).

model (MTP)	floor	score	wall	delivery mean	fan-out
Qwen3.6-27B-Omnimerge-v4 Q6_K (NextN self-spec)	15	99/100	2171 s	146 s	~7
″	5	99/100	2509 s	467 s	→22
″	1	98/100	2803 s	818 s	32 (max)
Gemma-4 26B-A4B-it Q4_K_M (assistant-MTP)	15	100/100	864 s	51 s	med 6
″	5	99/100	1090 s	145 s	med 14
″	1	95/100	1340 s	346 s	32 (max)
″ (cap 64)	25	100/100	773 s	31 s	~25
″ (cap 64)	20	100/100	823 s	38 s	~24
Gemma-4 26B-A4B-it F16 (assistant-MTP, cap 64)	25	100/100	1181 s	35 s	~32
″	20	100/100	1268 s	44 s	~31
″	15	100/100	1369 s	59 s	~30
″	5	100/100	1712 s	174 s	~34
″	1	100/100	2542 s	1282 s	68 (cap band)
Qwen3.6-35B-A3B UD-Q6_K (NextN self-spec, cap 64)	25	100/100	912 s	55 s	~21
″	20	100/100	890 s	64 s	~28
″	15	100/100	914 s	84 s	~24
″	5	100/100	1041 s	225 s	~37
″	1	100/100	1574 s	786 s	64 (cap)

Aggregate engine throughput stays pinned (Qwen ~75-80 tok/s regardless of concurrency), so equilibrium workers ≈ aggregate/floor. The A4B + assistant-MTP pair is ~2.5× faster end-to-end at every rung. All losses across all runs are agent-side (turn-budget wander on hard packages) — zero tool errors, zero LLM errors, engines clean throughout. The ab-pooled vs ab-naive pair isolates the SharedKVPool win at identical workload.

The A4B F16 ladder answers "what does weight precision cost under agentic load": same engine, same workload, F16 weights are ~1.5× slower end-to-end than Q4_K_M at every floor (bandwidth-bound MoE). At comfortable floors quality is saturated at Q4 (both 100/100), but at max fan-out precision shows: F16 floor 1 finished zero-loss at 68 workers grinding at ~1.1 tok/s median, while Q4_K_M at floor 1 loses 3-5 first-wave packages to turn-budget wander (95/100 original; a controlled re-run on the current StateGraph agent, same seed/cap, scored 97/100 with the identical loss signature — confirming the loss mode is model-precision-driven under cold-start contention, not a harness artifact).

The 35B-A3B ladder (worker cap raised to 64, floors extended to 20/25) is the cleanest full ladder: 100/100 at every floor including max fan-out, with the scheduler holding median per-turn tps on the floor (5→5.6, 15→17, 20→23, 25→28) as the fleet shrinks 64→21. It also locates the fan-out knee: wall time is flat (~890-1040 s) from floor 5 up, but floor 1 costs 1574 s — 64 workers oversubscribing 32 slots just queues (per-session tps 2.8, deliveries waiting 786 s mean). Sweet spot for this engine: floor 20-25 — same wall as floor 15 with the best per-package latency at ~21-28 workers.

---

Computing resources kindly provided by Vodafone