Impressive for its size, but total task latency is quite high
I am using your Q8_0 gguf on an RX 7900 XTX with Vulkan. I get ~2000 tok/s prefill with ~100 tok/s decode
As a first task I have told it to refactor all methods of one provider in an app ('provider' as in a class for wrapping API endpoints) into the provider of the parent route. This task was written in the file of the to-be-removed provider but the parent provider was not named directly.
To correctly finish this task the methods needed to be moved and all references needed to be updated by adjusting the imports to the parent providers file, aliasing the import and then replacing all occurences of the old class name with "import_alias.parentProviderClassName". Not the hardest task, but a good test in my opinion.
It did complete the work without any tool call errors, any syntax errors and without missing any necessary modification AND it even adhered to project customs (aliased style of importing other source files) - I find that impressive for that size.
High reasoning:
It needed 8min 40s for the entire task.
A large part was before it made any edits. It got confused about a helper method in the provider and thought about it for at least a minute.
Low reasoning:
It finished in 7min 30s. Nearly identical result as with high, but it did incorrectly forget to update a .md file (probably because the code analyzer does not flag this). It only told me that this file is now partly stale in its summary but did not update it itself. Interestingly, it reached the finished state after just 3-4 min and then spent the remaining time grepping through git log and fighting with the project-custom linter (which reported something outside its scope of edits).
- Both reasoning runs had the exact same source code at task start. So the run on high reasoning had the same (possibly confusing) linter output.
- It ran in pi. It had tools for analyzing source code available, so it did not need to reason about the syntax being correct or not (and didn't do it, it correctly relied on the tools after having edited files).
- Both runs used up ~50% of the 256K token window. However: I did load the entire code of the feature in before starting any prompt. Thats 30% already. I do this because I noticed that agents are way faster since they already know the code.
- The time measurements were started after system prompt, loaded source code etc. had been prefilled. Its the pure time for the task itself.
For comparison: Qwen 3.8 27B Swift 1.0 on medium took 3min 10s for the same task. It runs at 220 tok/s prefill and 40 tok/s decode.
I think you can improve Spark quite a lot if you manage to finetune it to yap a little less and be a bit more sure of itself 😄
Thanks for the detailed feedback! The overthinking and repeated checks are useful signals for improving efficiency, and we’re actively working on improvements in this area.