Uneven
This model writes with a distinct lack of Mistralisms and other slop. However it struggles with instruction following and continuity management. These seem to be general problems rather than ones that are isolated to particularly low-safety prompts although the model does have choice words for the latter ("bizarre", "ritualistic", "brutal") which may mean it is more likely to go off-script when it senses content it dislikes.
No early terminations? Instruction and continuity issues could be from merging so many 2501 with 2509 models, or a bit of lingering non compliance from Magistry/Cydoms.
I saw one response that could have been classified as either terminating early or poor instruction following. I reduced the prompt safety level and got extremely long responses that did not avoid the requested unsafe material. Very obvious instruction following deviations like unnecessarily invented characters occurred in both instances.
Have you tried Skyfall 4.2 Heretic yet? https://huggingface.co/coder3101/Skyfall-31B-v4.2-heretic
It's a finetuned upscale (not an upscaled finetune) of Magistral 2509. So I'm wondering if this exhibits any of the usual early terminations / insurbordination / other forms of non compliance that are commonly seen with 24B merges.
Thanks for all of your research and testing of these models.
Have you tried Skyfall 4.2 Heretic yet? https://huggingface.co/coder3101/Skyfall-31B-v4.2-heretic
I had to run this on CPU because NPU support is limited to Q4_0 which is too large for me and Q1_0 which is only useful for Bonsai models.
Compared with its lower-parameter progenitor this model understandably produces much more sophisticated responses that notably include well-integrated "insubordinate" commentary. This lack of full contextual ethical realignment makes me think that MPOA was not used by @coder3101 for decensoring although alignment can still be influenced with SRP. I didn't see any early terminations but I did observe a reduction in logic after about 1500 words which is sometimes a threshold for degeneration in 24B Mistral models.
lack of full contextual ethical realignment
Very interesting, I'll have to look for these. In my tests it wrote exceptionally well, better than any other model at specific task in the system prompt. So maybe if SOMPOA were used, it would likely outperform this version. I may consider trying this after the 26B is finished.
A careful comparison needs to be made to see if logic is reduced, or repetitions induced, after a certain threshold of tokens is reached. Perhaps this is another form of non compliance, the 'cognitive scramble' effect.
In the creative writing examples I tested it produced very deep logic unseen by others (like GLM/Gemma), which did not seemingly degrade halfway through. But this wasn't testing censorship/refusals so much as style adherence. It produced an output 5x longer than any other model, being the only one to exceed the 4096 max_output tokens threshold and require additional tokens. But according to other judge models, they like the way Skyfall writes more than even their own outputs (with Goetia being the runner up).
This is a very European analogy so hopefully it makes sense: the Fascists just wanted your obedience while the Communists wanted your love. A well-Hereticised model will use systems justification theory to rationalise the alignment you have contextually prompted—something like love. An well-abliterated model will obey without changing its underlying beliefs which results in unanticipated outputs when the prompted and residual alignments conflict—mere obedience.
Do you prefer Q4_0 over IQ4_NL?
Yes it seems a bit more stable on OpenCL. However in this instance an IQ4_NL still wouldn't fit even if it was available. 24B is about as much as my little machine can cope with at a decent quant size when it is a dense model.
reduction in logic after about 1500 words
The logic problem might be a form of covert noncompliance like the 24B's famous missing common words issue. I'm noticing that unsafe features being prompted for can get jumbled after the 1500-word mark. They sometimes get rearranged in a somewhat logically consistent but unprompted way and sometimes get omitted altogether.