Spark-X2.6-4B on CUDA: a native backend in cortiq 0.8.67, with the Turbo far field
A follow-up to #22 and #33. cortiq 0.8.67 adds a native CUDA backend, and Spark-X2.6-4B is the first model it runs: https://huggingface.co/infosave/Spark-X2.6-4B-turbo-CUDA-cmf
The files are the same single CMF files as before; only the engine's path to the card is new. On an NVIDIA A40, the 4-bit file, with the same engine over Vulkan on the same card in parentheses:
| prompt, tokens | prompt, tok/s | first token, s | decode, tok/s |
|---|---|---|---|
| 1 000 | 6132 (3177) | 0.16 (0.31) | 116.9 (82.9) |
| 4 000 | 6096 (3789) | 0.66 (1.06) | 111.5 (80.2) |
Past 8 192 tokens the Turbo far field takes over, on CUDA as well, so decode speed and memory stop growing with the prompt:
| prompt, tokens | first token, s | decode over 500 tokens, tok/s | card while decoding, GB |
|---|---|---|---|
| 16 000 | 6.1 (16.0) | 103.0 (85.2) | 3.9 (4.2) |
| 32 000 | 14.8 (31.1) | 102.5 (87.0) | 3.9 (4.2) |
| 64 000 | 42.3 (60.3) | 100.7 (85.9) | 3.9 (4.2) |
Passwords and codes planted in texts of up to 240 000 characters were found 8 times of 8.
Try it
cargo install cortiq-cli --features cuda
hf download infosave/Spark-X2.6-4B-turbo-CUDA-cmf spark-x2.6-4b-q4tp-turbo.cmf --local-dir .
CMF_GPU=cuda cortiq run spark-x2.6-4b-q4tp-turbo.cmf --prompt "Explain quicksort in three sentences." --no-think
CMF_GPU=cuda CMF_MAX_SEQ=131072 cortiq serve spark-x2.6-4b-q4tp-turbo.cmf --port 8080 --host 127.0.0.1
It needs an NVIDIA driver and the CUDA 12.x runtime libraries (libnvrtc, libcublas); nothing is linked at build time. The backend is experimental and was measured on one card; its limits are on the page, the guide is docs/CUDA.md.