EschaLabs/escha-runtime-qwen3moe

Escha Runtime — qwen3moe

By Escha Labs Inc.

Serving runtimes for Escha 2-/3-bit (eschamoe) quantized models of the qwen3_5_moe architecture (Qwen3.5 / Qwen3.6 Mixture-of-Experts, 256 experts). One repo per model architecture, one directory per engine — pick the engine that fits your workload:

	SGLang — sglang/	ZML — zml/
Best for	servers & teams	one user, one stream, zero-dependency deploys
Concurrency	continuous batching, paged KV, radix prefix cache	one request at a time
Single-user decode (4090)[^grid]	218–231 tok/s (flat in output length)	235–241 tok/s on ≥500-token answers (+8–14%); ~190 tok/s on ≤128-token replies
TTFT, 128→1890-tok prompt[^grid]	0.06–0.24 s	0.06–0.35 s
Max context (4090)[^ctx]	up to 159k tok @ MEM=0.78, 235k @ 0.90 (ships CTXLEN=32768)	262,144 @ ESCHA_MEM=0.93 (ships ESCHA_CTX=1024)
Multi-turn prompt reuse	radix cache (branching, cross-session)	append-only, single session
Tool calls / JSON schema / thinking parser	yes	no
Sampling (temperature>0)	full speed	~104 tok/s (fast path is greedy-only)
Install	Python 3.12 venv + CUDA-12 PyTorch	one binary, no Python, no CUDA toolkit

Neither engine is uniformly faster. ZML wins sustained decode on long single-stream answers and is far simpler to deploy; SGLang wins short replies, wins time-to-first-token on long prompts, and is the only option for concurrency, tool calling or structured output. If unsure: multi-user, agents-at-scale, or structured output → SGLang; long-form single-user generation on your own GPU → ZML.

The ZML engine is built on ZML (Zig + MLIR/XLA); the SGLang engine on a fork of SGLang. Both run the same Escha CUDA kernels and serve the same model files.

[^grid]: Same-harness measurement, 2026-07-26: one RTX 4090, the same model files, the same streaming OpenAI client, greedy, batch 1, medians of 2 reps over an ISL×OSL grid (128–1890 in, 128–1890 out). Decode = steady-state tokens/s between the first and last streamed token. Per-cell decode: SGLang 176.8–230.9 (median 218.2), ZML 184.6–240.7 (median 234.8); ZML leads every cell with ≥256 output tokens and trails on 128-token replies, where its 16-token fused decode chunk dominates. TTFT by input length (ZML / SGLang): 128 tok 55/55 ms · 500 tok 95/91 ms · 1000 tok 178/143 ms · 1890 tok 353/237 ms.

[^ctx]: Measured 2026-07-26 on one RTX 4090 (24 GB) with Qwen3.6-35B-A3B-Escha-W2. Both engines ship a conservative default context and let you raise it. SGLang reports its KV pool at startup; launched with CTXLEN=262144 it allocated 159,480 tokens at its shipped MEM=0.78 and 234,796 at MEM=0.90 — that pool size is the ceiling for a single sequence. ZML serves a 261,966-token prompt at ESCHA_CTX=262144. Both are memory-bound near the cap, not architecture-bound: KV on this hybrid model is only ~20 KiB per token (10 attention layers; the 30 gated-delta-net layers hold a fixed ~66 MB recurrent state), so 262,144 tokens is 5.37 GB beside 12.3 GB of weights. In fairness to SGLang: its 234,796-token pool at MEM=0.90 still left 1.38 GB of VRAM free, so it would very likely also reach the 262,144 cap at a higher mem-fraction-static — we did not test that, and 0.78 is simply the conservative default the runtime ships. Read this row as "both engines reach the model's native context on a 24 GB card", not as a ZML advantage.

Compatible models

Model repo	Bits
EschaLabs/Qwen3.6-35B-A3B-Escha-W2	2-bit (eschamoe)

These runtimes target qwen3_5_moe. A model of a different architecture will not load — use the matching escha-runtime-<arch> repo.

Quickstart

SGLang engine (full detail: sglang/INSTALL.md, incl. the per-GPU cookbook):

python3.12 -m venv .venv && source .venv/bin/activate
pip install -U pip wheel
pip install "torch==2.9.*" --index-url https://download.pytorch.org/whl/cu128   # cu12 torch first
pip install ./sglang/escha-*.whl   # pulls the bundled sglang fork + its full dep closure

hf download EschaLabs/Qwen3.6-35B-A3B-Escha-W2 --local-dir ./Qwen3.6-35B-A3B-Escha-W2
MODEL=./Qwen3.6-35B-A3B-Escha-W2 bash sglang/serve.sh

ZML engine (full detail: zml/INSTALL.md) — no Python at all:

tar xf zml/escha-zml-serve-*-linux-x86_64.tar.gz && cd escha-zml-serve-*
hf download EschaLabs/Qwen3.6-35B-A3B-Escha-W2 --local-dir ./model
./escha-serve

Long context and agentic use (ZML engine)

ESCHA_CTX reaches the model's native 262,144 tokens. Measured on one 4090 with facts planted at 10%, 50% and 90% depth and all three asked for at the end — the check that positions and recurrent state thread correctly across the whole prompt:

prompt	prefill	rate	recall
15,066 tok	5.3 s	2,858 tok/s	3/3
60,066 tok	24.9 s	2,411 tok/s	3/3
129,966 tok	79.0 s	1,646 tok/s	3/3
261,966 tok	259.1 s	1,011 tok/s	3/3

Prefill is ~N²/2 work, so the rate falls with length. The comfortable band on a 24 GB card is ~8k–130k, where TTFT is seconds; 262k answers correctly but takes ~4.3 minutes to read.

Multi-turn: a conversation whose prompt grows each turn reuses the previous prefix. On a 14k-token conversation the first turn takes 4.11 s and each following turn 1.27 s (~99.8% of the prompt reused). Reuse requires an exact append — editing earlier history re-reads the prompt — and only one conversation is cached. SGLang's radix cache is faster in absolute terms here and handles branching and multiple sessions; the ZML cache exists to make single-session agent loops practical, not to match it.

Decode slows as position grows on both engines, because decode attention is O(position). On ZML, measured with prefill excluded: 227 tok/s at position ~1k, 163 at 3k, 124 at 7.5k. (That measurement divides total request time and so reads ~5% below the streamed grid above — method, not configuration.)

Requirements (shared)

• NVIDIA GPU, compute capability 8.0–12.0 (Ampere → Blackwell), Linux x86-64. Kernel launch route auto-selects per GPU. Per-architecture and per-VRAM launch recipes: sglang/INSTALL.md → Running on your GPU.
• SGLang engine: Python 3.12 + CUDA-12 PyTorch (the wheel handles every other dependency). 24 GB VRAM recommended (16 GB works with reduced context).
• ZML engine: no runtime dependencies — the archive bundles the CUDA runtime; just an NVIDIA driver (R550+) and glibc 2.27+ (RHEL/Rocky/Alma 8 and newer). 24 GB VRAM; one request at a time.

Known limitations

• DETERMINISTIC=1 fails on consumer Blackwell (sm_120). The deterministic attention kernel requests 104 KB of shared memory per block, above the sm_120 limit, and the server exits during startup. It works on Ampere, Ada and Hopper. Greedy output is in any case not bit-reproducible across requests on either engine — batch composition changes fp16 accumulation order, so a near-tie can flip and a long reasoning chain diverges from there.
• The ZML engine serves no /v1/completions (HTTP 404). Use /v1/chat/completions.
• The ZML engine does not validate requests. An unknown model field is served anyway instead of returning model_not_found, and usage.prompt_tokens is wrong when a prompt is truncated.
• The ZML engine can wedge on an over-long prompt on a 16 GB card, requiring kill -9 rather than returning an error. This is why it asks for 24 GB — use the SGLang engine on 16 GB, where the same prompt returns a clean HTTP 400.

License

Everything here is released under the Apache License, Version 2.0 — see LICENSE. All bundled third-party code is permissive (Apache-2.0 / MIT / BSD-3-Clause; NVIDIA runtime libraries in the ZML bundle under the NVIDIA EULA's redistributable-runtime terms) — no copyleft. Full texts and the component inventory: THIRD_PARTY_LICENSES/. Model weights are not in this repo and carry their own license in the model repository.