XHToken

/

Spark-X2.5-4B

Like 1.1k

Follow

SparkLLM 645

I forgot to say - Thank you @xhtoken team ! Its really an excellent model, that I will be using for actual work. Look forward to your future work !

I built the llama.cpp fork and tested out this model. Extremely good quality code generation ! On a mini laptop with no dedicated GPU (integrated 780M) @15W ,i.e. a low power system, I am getting 12-15TPS decode, which is incredible. With a normal dGPU like a 3060, I would assume you'd get 80TPS

I 99% use Qwen 3.8 27B locally and Qwen 3.8 Max cloud for my daily coding work

I have to say this 4B model really surprised me with the coding quality. I won't mind using this instead of Qwen 27B for less complex work

I would suggest to the team to make builds of their llama.cpp fork for the most common releases - vulkan, cpu, CUDA, ROCM, for Ubuntu and Windows. Its not difficult, and it would make it very fast and convenient for users to quickly test your models...

Hi, you said you are on 780M laptop, but how are you using Qwen 27b on that? or you have other computer?

I also have a 780M / 8845HS (60W) and I’m running a 27B model with MTP at around 12 tok/s. It’s possible, but I prefer MoE models instead.

I also have a 780M / 8845HS (60W) and I’m running a 27B model with MTP at around 12 tok/s. It’s possible, but I prefer MoE models instead.

it seems that APU is powerful for AI rather than 4gb VRAM + RAM offloading, because I cant run dense models bigger than 9b, and MoEs only at 15t/s

I built the llama.cpp fork and tested out this model. Extremely good quality code generation ! On a mini laptop with no dedicated GPU (integrated 780M) @15W ,i.e. a low power system, I am getting 12-15TPS decode, which is incredible. With a normal dGPU like a 3060, I would assume you'd get 80TPS

I 99% use Qwen 3.8 27B locally and Qwen 3.8 Max cloud for my daily coding work

I have to say this 4B model really surprised me with the coding quality. I won't mind using this instead of Qwen 27B for less complex work

I would suggest to the team to make builds of their llama.cpp fork for the most common releases - vulkan, cpu, CUDA, ROCM, for Ubuntu and Windows. Its not difficult, and it would make it very fast and convenient for users to quickly test your models...

Hi, you said you are on 780M laptop, but how are you using Qwen 27b on that? or you have other computer?

780M's have a feature most people don't know. In Linux you can assign giant amounts of RAM to them as UMA. It behaves like a low end GPU with very large VRAM. They use it much faster than CPU offload.

On my Laptop I have 32GB RAM, I load models upto 29GB before it causes a kernel panic. On my Desktop I have 128GB RAM and a 780M, I can easily run Deepseek Flash or GLM Flash, upto 124GB models. Speeds are pretty decent, MOE models ~35B 25Tok/s at low power usage - 18Watts. Deepseek et all run >10 tok/s on the desktop

Similar to Halo Strix AI+ 395 boxes, DGX Spark or Mac Mini (upto 512GB). Memory speed differs in each case, but all of them use normal RAM instead of high speed VRAM.

On my Laptop I have 32GB RAM, I load models upto 29GB before it causes a kernel panic. On my Desktop I have 128GB RAM and a 780M, I can easily run Deepseek Flash or GLM Flash, upto 124GB models. Speeds are pretty decent, MOE models ~35B 25Tok/s at low power usage - 18Watts. Deepseek et all run >10 tok/s on the desktop

Similar to Halo Strix AI+ 395 boxes, DGX Spark or Mac Mini (upto 512GB). Memory speed differs in each case, but all of them use normal RAM instead of high speed VRAM.

that is great, thought of buying laptop with 780m, dont need for gaming much, but seeing that it is great for AI that allows to use so big dense models, gave new interest, but cant buy it now, and dont really need AI at all. But it is thing to be considered really, because this APU is somewhat cheap unlike Halo or 890M, and if you say it runs Dense 27b model at 12 tokens then it is total blast for such price.

I also have a 780M / 8845HS (60W) and I’m running a 27B model with MTP at around 12 tok/s. It’s possible, but I prefer MoE models instead.

it seems that APU is powerful for AI rather than 4gb VRAM + RAM offloading, because I cant run dense models bigger than 9b, and MoEs only at 15t/s

You are making some mistakes perhaps. 35B MOEs like Qwen 3.6, on my mini laptop w 780M runs quite easily at 25Tok/s at very low power settings to save battery - 18W.
Dense models like Qwen 3.8 27B at the same power runs at 15-12 Tok/s

Here is a sample command I use for running a compressed 3.6 35B for coding:

export LLAMA_ARG_GRP_ATTN_N=4 && export LLAMA_ARG_GRP_ATTN_W=4096 &&
./llama/llama-server
-m ./GGUF/Qwen3.6-35B-A3B-UD-Q3_K_XL-REAP-RangerX.gguf
--host 0.0.0.0 --port 8080
--no-mmap --flash-attn 1 --jinja -b 128 -ub 128
--chat-template-kwargs '{"enable_thinking": false}'
--spec-type draft-mtp,ngram-simple --spec-draft-n-max 4 --spec-ngram-simple-size-n 12 -n -1
--ctx-size 35526 --cache-type-k q5_0 --cache-type-v q5_0 --cache-ram 0 --ctx-checkpoints 16
--seed 3407 --prio 2 --temp 1.0 --repeat-penalty 1.0
--presence_penalty 1.5 --min-p 0.0 --top-k 20 --top-p 0.95

I also have a 780M / 8845HS (60W) and I’m running a 27B model with MTP at around 12 tok/s. It’s possible, but I prefer MoE models instead.

it seems that APU is powerful for AI rather than 4gb VRAM + RAM offloading, because I cant run dense models bigger than 9b, and MoEs only at 15t/s

You are making some mistakes perhaps. 35B MOEs like Qwen 3.6, on my mini laptop w 780M runs quite easily at 25Tok/s at very low power settings to save battery - 18W.
Dense models like Qwen 3.8 27B at the same power runs at 15-12 Tok/s

Here is a sample command I use for running a compressed 3.6 35B for coding:

export LLAMA_ARG_GRP_ATTN_N=4 && export LLAMA_ARG_GRP_ATTN_W=4096 &&
./llama/llama-server
-m ./GGUF/Qwen3.6-35B-A3B-UD-Q3_K_XL-REAP-RangerX.gguf
--host 0.0.0.0 --port 8080
--no-mmap --flash-attn 1 --jinja -b 128 -ub 128
--chat-template-kwargs '{"enable_thinking": false}'
--spec-type draft-mtp,ngram-simple --spec-draft-n-max 4 --spec-ngram-simple-size-n 12 -n -1
--ctx-size 35526 --cache-type-k q5_0 --cache-type-v q5_0 --cache-ram 0 --ctx-checkpoints 16
--seed 3407 --prio 2 --temp 1.0 --repeat-penalty 1.0
--presence_penalty 1.5 --min-p 0.0 --top-k 20 --top-p 0.95

this Qwen is not normal Qwen, it is tightly quantized, and it is even Reap, which is not full one. Also you usint MTP, which doesnt work on partial offloading normally or at all. Wonder, which version of Qwen 27b you using?(Wonder why REAP model? as far as tested, all of them are broken and loop too often)

@theodoreH
The REAP, quantization etc do not increase the speed, they only reduce the memory size. In my case, I am specifically using coding agents, and this setup is optimized for that

For 27B Ridge is a 3.7bpw quant that is really good at coding

MTP is hit and miss. For DGPUs it works 100%, for 780Ms I see 5-10% speedup for MOE models and negative performance for dense models

I did not have looping problems with the REAP models I use right now. Good REAP models work fine, e.g. 0xSero is a very experienced REAP builder

I also have a 780M / 8845HS (60W) and I’m running a 27B model with MTP at around 12 tok/s. It’s possible, but I prefer MoE models instead.

it seems that APU is powerful for AI rather than 4gb VRAM + RAM offloading, because I cant run dense models bigger than 9b, and MoEs only at 15t/s

You are making some mistakes perhaps. 35B MOEs like Qwen 3.6, on my mini laptop w 780M runs quite easily at 25Tok/s at very low power settings to save battery - 18W.
Dense models like Qwen 3.8 27B at the same power runs at 15-12 Tok/s

Here is a sample command I use for running a compressed 3.6 35B for coding:

export LLAMA_ARG_GRP_ATTN_N=4 && export LLAMA_ARG_GRP_ATTN_W=4096 &&
./llama/llama-server
-m ./GGUF/Qwen3.6-35B-A3B-UD-Q3_K_XL-REAP-RangerX.gguf
--host 0.0.0.0 --port 8080
--no-mmap --flash-attn 1 --jinja -b 128 -ub 128
--chat-template-kwargs '{"enable_thinking": false}'
--spec-type draft-mtp,ngram-simple --spec-draft-n-max 4 --spec-ngram-simple-size-n 12 -n -1
--ctx-size 35526 --cache-type-k q5_0 --cache-type-v q5_0 --cache-ram 0 --ctx-checkpoints 16
--seed 3407 --prio 2 --temp 1.0 --repeat-penalty 1.0
--presence_penalty 1.5 --min-p 0.0 --top-k 20 --top-p 0.95

I love finding little tweaks that improve my setup - I've got a tiny little mini-pc with a 580m APU and adding your ngram-simple gets me up to 4.5 t/s for Qwen 3.8 27B (Empero-ai's Ridge quant) on predictable work. Unfortunately, the 580m is missing a bunch of the mat-mul enhancements in later RDNA versions, so I don't think I can get much more out of it, but I'm curious if you do any other tweaks for the dense model vs the MOE?

I also have a 780M / 8845HS (60W) and I’m running a 27B model with MTP at around 12 tok/s. It’s possible, but I prefer MoE models instead.

it seems that APU is powerful for AI rather than 4gb VRAM + RAM offloading, because I cant run dense models bigger than 9b, and MoEs only at 15t/s

You are making some mistakes perhaps. 35B MOEs like Qwen 3.6, on my mini laptop w 780M runs quite easily at 25Tok/s at very low power settings to save battery - 18W.
Dense models like Qwen 3.8 27B at the same power runs at 15-12 Tok/s

Here is a sample command I use for running a compressed 3.6 35B for coding:

export LLAMA_ARG_GRP_ATTN_N=4 && export LLAMA_ARG_GRP_ATTN_W=4096 &&
./llama/llama-server
-m ./GGUF/Qwen3.6-35B-A3B-UD-Q3_K_XL-REAP-RangerX.gguf
--host 0.0.0.0 --port 8080
--no-mmap --flash-attn 1 --jinja -b 128 -ub 128
--chat-template-kwargs '{"enable_thinking": false}'
--spec-type draft-mtp,ngram-simple --spec-draft-n-max 4 --spec-ngram-simple-size-n 12 -n -1
--ctx-size 35526 --cache-type-k q5_0 --cache-type-v q5_0 --cache-ram 0 --ctx-checkpoints 16
--seed 3407 --prio 2 --temp 1.0 --repeat-penalty 1.0
--presence_penalty 1.5 --min-p 0.0 --top-k 20 --top-p 0.95

I love finding little tweaks that improve my setup - I've got a tiny little mini-pc with a 580m APU and adding your ngram-simple gets me up to 4.5 t/s for Qwen 3.8 27B (Empero-ai's Ridge quant) on predictable work. Unfortunately, the 580m is missing a bunch of the mat-mul enhancements in later RDNA versions, so I don't think I can get much more out of it, but I'm curious if you do any other tweaks for the dense model vs the MOE?

Hi @bigjeff5

The main use for that ngram-simple tweak is a massive speedup when it comes to coding.If it sees a large script of code, where it only has to make few changes but most of the script stays the same, its able to quickly output the parts that are still the same (self decoding)

The main thing I do is look for better models for my use case - coding. Spark does such an amazing job, its really similar to Qwen 3.6 35B, at 1/3 the size

For MOE models, I think most tweaks are around making sure the experts you use most stay in VRAM. But in your case it doesn't apply since you are using system RAM. I assume you are putting the whole model in UMA ? If you are doing CPU offload, you should rather assign more memory to the 580m (only possible in Linux)

Qwen 3.8 27B is the smartest coding model all the way till 120B, so you just have to accept that it will run slow. If you want faster models you have to drop to less intelligent ones

FWIW Qwen 3.6 27B is pretty much equally good, but decently faster