Premise
So this is a basic slerp merge between a smart model and a good prose model. Prose and smarts. What we all want in an uncensored RP model right? I feel like Solar has untapped potential, in any case.
Sao10K's Frostwind finetune is a key component of the mixture, its smarts are impressive. NyxKrage's Frostmaid experiment, which merges Frostwind with a frankenmerge of Noromaid and a mystery medical model, delivers quite impressive prose. His model creatively incorporates long-range context and instructions too, despite being slightly incoherent due to the fraken merging.
So those are the main ingredients. Thanks to Nyx for sorting out the pytorch files btw.
Recipe
So, the recipe. I basically just gradient SLERP'd Frostwind into Frostmaid with these params:
- filter: self_attn
value: [0.9, 0.6, 0.3, 0, 0]
- filter: mlp value: [0.3, 0.6]
- value: 0.5 # fallback for rest of tensors
Tentative Dozen or So Test Conclusion
This made a model that was actually pretty much everything I was looking for - NEARLY as smart as Frostwind but with MOST of Frostmaids punchy prose. I tried doing TIES merges and DARE ties merges, but they actually came out worse, because both models have major weaknesses - one is very dry and gpt-ish, the other is a little loose with what's going on. The ties merges tended to bring out those qualities, dare even worse. So I stuck with this. It's not AS smart as Frostwind, so you maybe have to regen a little, but it's pretty smart, and quite creative. A sweet spot hopefully. Maybe someone merge wiser than I can do more with this recipe, but I'm very pleased with it, it did what I was hoping for - a smaller model I can mobile dgpu and produces pretty outsized quality responses (it's fairly zealous tho, be warned). I've only played with it a TINY bit, so there may be qualities or flaws I've missed.
Cheers to all the finetuners, mergers and developers without which open source models wouldn't be half of what they are.
Resources used:
https://huggingface.co/NyxKrage/FrostMaid-10.7B-TESTING-pt
- Downloads last month
- 1