EschaLabs

/

escha-runtime-qwen3dense

Like 13

Follow

Escha Labs 235

Follow-up on the same issue, with a concrete case for what the 0.5.17 cap costs on hybrid-SSM (GDN) deployments.

Two upstream changes landed after 0.5.17 that matter specifically for the W2 dense build on small cards:

• #15829 ([feat] Support extra_buffer in Mamba2-based models) introduced --mamba-track-interval, decoupling where Mamba state checkpoints are taken from chunked_prefill_size.
• #34763 ([Spec] Support mamba-radix-cache-strategy extra_buffer_lazy with DFLASH) added --mamba-radix-cache-strategy extra_buffer_lazy, which plans track slots lazily and now works with speculative decoding.

In the shipped tree, checkpoints are only produced at chunk boundaries, so the prefix-cache capacity condition is effectively

required mamba slots ≈ kv_pool_tokens / chunked_prefill_size ≤ max_mamba_cache_size

On 2× consumer 12 GB cards (sm_120), Qwen3.8-27B-Escha-W2, --kv-cache-dtype int2, 196,608-token context, --max-total-tokens 220000, --mamba-ssm-dtype float16, --mamba-scheduler-strategy extra_buffer, this is what the trade-off measures:

So on this class of hardware the only way to keep the prefix cache alive is to raise chunked_prefill_size, and that costs 6.5× on TTFT under concurrency and 4× on long-prefill wall time. With --mamba-track-interval the two knobs are independent, which removes the trade-off entirely rather than trading one regression for another.

As an interim workaround we backported upstream's --enable-int8-mamba-checkpoint into the installed tree (int8 checkpoint slots, ~2× capacity at the same VRAM) so the capacity condition can be met at CHUNK=4096. It works around the symptom, but the track-interval change is the actual fix, and it only ships in ≥ 0.5.18 — which needs torch 2.13, hence this request.

Two questions:

1. Is a wheel built against torch 2.13 (or a rebase onto SGLang ≥ 0.5.18) on the roadmap, even roughly?
2. In the MoE repo (discussion #8) there was a note about open-sourcing the whole runtime stack. If that includes the SGLang fork, downstream users could rebase onto current upstream themselves and stop waiting on wheel rebuilds — is that still the plan?

Happy to test any preview build on sm_120 and report back with the same measurements.

Thanks for both reports. Pushed 1.2.2. See the updated README/INSTALL.md for the torch-version table and what changed.

torch 2.13: not built or tested yet (our box tops out at CUDA 12.8, 2.13 wants 12.9/13.0). The stable-ABI design should extend there. It never touches TensorImpl fields or hardcodes DispatchKey ordinals, so none of your three break points apply. But that's unverified.

mamba-track-interval: It is already in the bundled fork (an undocumented oversight on our end) and decouples checkpoint spacing from chunked_prefill_size under --mamba-scheduler-strategy extra_buffer, which your command already sets. Pass it straight to serve.sh; curious whether it removes the trade-off on your 2×12GB setup.

Open-sourcing the fork: still the plan, no date yet. We will try to release the full runtime stack before November, and it will be opensourced together with our next major release. Stay tuned!

Thank you — 1.2.2 stable ABI is exactly what was needed.

Turnaround on this was remarkably fast, and the stable-ABI approach solves the problem at the root rather than per-torch wheels. Much appreciated.

One offer: your 1.2.2 verification was on an RTX 4090 (sm_89). I'm running Qwen3.8-27B-Escha-W2 on 2x RTX 5070 12GB (sm_120, consumer Blackwell) with TP2 + OSCAR int2 KV, which is a tighter memory regime than most setups — mamba state is 75.5 MB/request (48 linear-attention layers x 48 value heads x 128 x 128, fp16), so the mamba pool takes ~4x the space of the int2 KV pool.

I'll test 1.2.2 with torch 2.13 + sglang 0.5.18 on sm_120 and report back: import, byte-identical greedy output vs 1.2.1, and decode throughput. If sglang-kernel from the CUDA-12 index behaves differently on sm_120 than sm_100, that seems worth documenting in INSTALL.md.

Re: --mamba-track-interval already working under --mamba-scheduler-strategy extra_buffer — thank you for the correction, I had assumed it needed 0.5.18. I'll measure it on the current build.

Looking forward to the runtime OSS release.

Report: 1.2.2 runs on torch 2.13 / sm_120 — thank you

The stable-ABI build works. I have been running it on torch 2.13.0+cu129 with stock upstream sglang 0.5.19 (RTX 5070 12GB x2, sm_120, WSL2, TP2) for a full day now, and it is serving production traffic as of today.

To be precise about what this unlocked: the stable ABI did not make anything faster by itself — it made the engine choosable. Before 1.2.2 we were pinned to torch 2.9 and therefore to sglang <= 0.5.17. Now the same 2-bit weights run on 0.5.19, which is where the newer mamba/radix features live.

Measured on Qwen3.8-27B-Escha-W2 with OSCAR INT2 KV, 12GB x2:

Quality is unchanged from our 0.5.15 deployment: 19/20 on a discriminative long-context test (two-field extraction among near-identical distractors, multi-position aggregation, ordered enumeration).

One small thing: the model card still says pip install 'torch==2.9.*' is "a hard pin, not a suggestion". That is no longer true for 1.2.2 — it may be worth updating, since the 2.9 pin is exactly what stops people from reaching 0.5.19.

Full configuration, patches and numbers, in case they are useful to other 12GB users: https://github.com/Nakanokensetsu/sglang -> escha_oscar_0519/

Thanks again for turning this around so quickly.