unsloth/Qwen3.8-27B-NVFP4 vs. Inferact/Qwen3.8-27B-NVFP4?
Let's gather everyone's experiences/benchmarks here.
I've not tested in a lighter harness, but in both hermes and oh-my-pi, when expected to take action, this model very easily get's caught in a thinking loop. Enough context is gathered, but the model continues to rehearse what should be done, but never takes action. Not always mitigated by medium setting.
I suspect the quantization method is at fault. I've not seen this behavior with 3.6 27B or any other model larger than 12B. Reasoning remains coherent but tasks where there is the potential for mistakes cause inaction. These tasks were in the feature engineering and Linux system admin domains.
I experienced the same. I also tried different combinations of changing repetition penalty and presence penalty on top of changing the chat template Jinja file. No luck. Unsloth version is the same.
I've not tested in a lighter harness, but in both
hermesandoh-my-pi, when expected to take action, this model very easily get's caught in a thinking loop. Enough context is gathered, but the model continues to rehearse what should be done, but never takes action. Not always mitigated bymediumsetting.I suspect the quantization method is at fault. I've not seen this behavior with 3.6 27B or any other model larger than 12B. Reasoning remains coherent but tasks where there is the potential for mistakes cause inaction. These tasks were in the feature engineering and Linux system admin domains.
I experienced the same issue. But, it decides to do what it needs to do, and does it, eventually. 😄
FYI, This happens to me even with the official FP8 quant from Qwen team.
Sometimes it loops tool calls, sometimes it swaps tool names with skill script names, sometimes it generates gibberish thinking traces with characters from various languages.
FYI, This happens to me even with the official FP8 quant from Qwen team.
Sometimes it loops tool calls, sometimes it swaps tool names with skill script names, sometimes it generates gibberish thinking traces with characters from various languages.
Have you experienced this with the fp16 weights?
I experienced the same. I also tried different combinations of changing repetition penalty and presence penalty on top of changing the chat template Jinja file. No luck. Unsloth version is the same.
I have not reattempted with nvfp4, but fp8 seems to be highly sensitive to the generation config. Whatever Qwen shipped is not their own recommended config for thinking mode.
What engines are you all using? I've only tried vLLM + MTP. I'm going to try SGLang + DSpark as I've seen some posts mentioning success with NVFP4.
Anecdotally, reasoning seems to reflect a very detailed world model and an intricate self-verification. Absurdly long reasoning traces might be a signature with this model, but unlike past releases the final output is successful. Outdated information and nuance seem to be the trigger for loops. For example, the model completely adjudicated whether to comply with my request because of a linux kernel feature added in version 5.