Now we have independent numbers
I don't believe this for a second. This model does not have a legitimate intelligence score of 52. I assure you that across the board it's much weaker than the other much larger models with this approximate score, and if it ever gets on Arena it will be ranked far lower across all categories, and likely lower than even Qwen 3.5 27b.
This community doesn't have enough skepticism and critical thinking skills. Alibaba has a long history of pandering to coders (the most vocal users of OS models) while inflating test scores across the board with various test maxing techniques. So in the end they appear to be absurdly powerful, even their tiny 4b models, yet in real world use cases they're vastly inferior to other models with much lower test scores.
For example, Qwen 3.5 27b has higher test scores than Gemma 4 31b, but across all categories on the Arena it's ranked far lower. And for good reason. Not only does it burn through far more tokens when thinking, enters more infinite loops, and so on, but the complex outputs are shit by comparison. For example, the stories are far more repetitive and filled with blatant contradictions to itself and the user's prompt. Same thing with their tiny models like 4b. They have test scores rivaling models more than 10x larger, but when you use them they're astonishingly bad. They struggle to even do simple things like writing synonym lists of common words. In short, starting with Qwen2.5 Alibaba's test scores have become progressively more disconnected from their real world performance.
It's not that this model is bad. It's really good at coding and agentic tasks, but only OK to very bad at most other tasks. I'll gladly eat my words if it does well on arena, but I doubt it will even beat Qwen 3.5 27b.
I mean that's the numbers from their benchmarks. I guess your tests are another part of the picture. But what I can tell is that I like to read the reasoning, and I find it really smart in comparison to the kind-of-smart 3.6-27B, I have much more pleasure to read it they are more interesting. But I'm wondering how the quality of the reasoning is affected by domain and by the query itself. I found it more confident, opinionated and surprising than 3.6. I know you are talking about the 3.5, so maybe not fair to compare considering your position on the 3.6...
I saw it fails on the simple https://huggingface.co/Qwen/Qwen3.8-27B/discussions/50 but I finally resigned myself judging that we actually really don't need it to know that natively since a model will always be ran on a machine with at least a shell.
I still need to test it a lot more on other domain than coding/reasoning
Hmm, on high it's thinking forever (which is no fun considering it's speed on my hw). On low it did quiet well on my synthetic benchmarks and my normal "Copilot Adventures Agent Mode"-test parcour. Very focused reasoning, tool calling and implementation (but made the same implementation errors Qwen3.6 35B A3B made and thought much longer how to fix them). Interesting though, after the tests passed and it only had to do a bit of documentation and UI, it decided to build a complete different application "weather forecast" UI 🙃 So much for instruction following.
So in the end, instead the working solution Qwen3.6-35B-A3B produced in 1 hour and 2M Tokens, it produced a half finished solution and one i didn't ask for in 5 hours and 3M Tokens on my M2Pro.
I think i'll stick with a hardened (KatCoder/Vision, froggeric chat-template) Qwen3.6-35B-A3B for now... (it's slutty but fast 😁 ) ... and wait for a Qwen3.8 MoE...
Yeah MOE would be awesome!
it decided to build a complete different application
I don't know how you are running the inference but can't it be because of context-shift? with llama-server it's --no-context-shiftif ever that's the cause
