Spark-X2.6-1.7B running in the browser on WebGPU (ONNX, official INT4 weights)
Hi XHToken team, thanks for Spark-X2.6!
I converted Spark-X2.6-1.7B to ONNX so it runs in the browser on WebGPU, with no server and nothing to install.
- Demo: https://huggingface.co/spaces/mohdmahmodi/Spark-X2.6-1.7B-WebGPU
- Model files: https://huggingface.co/mohdmahmodi/Spark-X2.6-1.7B-ONNX-WebGPU
- Source and validation: https://github.com/mohdmahmodi/Spark-X2.6-1.7B-onnx-webgpu
The 4-bit weights are your GPTQ codes from Spark-X2.6-1.7B-INT4, repacked for onnxruntime-web without changing them. Thinking modes and tool calling work, using the official chat template.
I compared it with the original model in transformers on the same prompts (greedy, thinking off):
| bf16 | official INT4 | this build | |
|---|---|---|---|
| GSM8K, 1,319 problems | 75.7% | 72.7% | 72.7% |
| BFCL v3 simple, 200 cases | 76.5% | 52.5% | 52.5% |
| Needle in a haystack, 1k to 16k tokens | 12/12 | 12/12 | 12/12 |
One thing you might want to look at: the official INT4 checkpoint scores 52.5% on BFCL against 76.5% for bf16, and that's with the INT4 checkpoint running in transformers, so it's not from my conversion. Most of the drop is the INT4 model answering in text where bf16 calls the tool. When it does call, the arguments are just as accurate.
On an RTX 3090 Ti in Chrome it does about 105 tok/s, and it stays at that speed up to 16k tokens of context. The download is 1.21 GB, the browser caches it, and it runs within the default WebGPU limits.
It also works with Transformers.js 4.3:
import { pipeline } from "@huggingface/transformers";
const generator = await pipeline("text-generation", "mohdmahmodi/Spark-X2.6-1.7B-ONNX-WebGPU", { device: "webgpu", dtype: "q4f16" });
const out = await generator([{ role: "user", content: "Hello!" }], {
max_new_tokens: 256, tokenizer_encode_kwargs: { enable_thinking: false },
});
console.log(out[0].generated_text.at(-1).content);
I've only tested it on Windows with an NVIDIA GPU so far. If anyone tries it on a Mac, an AMD or Intel GPU, or a phone, let me know how it goes. And if you think it's useful, I'd be glad to see it linked from the model card.
Thanks for the contribution and detailed validation! We’ve added a WebGPU section to our README linking to your demo, model files, and source code.