26B vs 31B
I tried both versions of Pantheon Reasoning 1.1. And I guess the difference between the two architectures is just THAT big, huh... Because, assuming you fed them the same data of thinking traces, they think NOTHING alike!
Where 31B has concise blocks at ~500 tokens, +/-100, 26B thinks FOREVER. What it does is it starts drafting the response and then double-checking itself. This will go for thousands of tokens and most of the time it just remains stuck in a recursive loop of "wait, but let me quadruple check..." For the first 2-3 iterations, the drafting does seem to be introducing improvements and sometimes it does catch critical issues, like misattributions of context or forgetting to acknowledge an important nuance, but after the first few revisions, the drafts just regress into mutations of themselves without any resolution in sight.
Keep in mind, I used it at Q6_K_L by bartowski and keep the kv cache uncompressed at bf16. So it's not a quant issue, but default model behavior.
Raising the rep penalty introduces its own bag of worms. I also tried to write some constraints into the system prompt, telling it that it can only draft for 3 iteration maximum, but for some reason, when you spell it out, it feels like it starts making the first draft "bad" on purpose so its thinking process "looks" like its iterating and improving something... It also doesn't reliably prevent the recursion problem from happening to begin with either...
Lowering temp from 1.0 to 0.7 also doesn't seem to have much of an effect.
Honestly, I'm a bit stumped at what else to try. The bigger 31B sibling works like a charm on the same system prompts, big or small, without having to introduce any explicit CoT instructions. Well, at least it was interesting to observe the sheer magnitude of difference in the two models.
I'll see if I can't take another pass at this in the near future, leveraging the fresh, much more functional StyleTune V2 as its base, since that worked well for 31B.