Saftetensors
Would be nice to have access to the full bf16 safetensors of this model for making custom quants and finetuning the model further.
Not to speak for drummer but why can't you use what's available?
And why would you want to finetune this model further? If you have the raw resources to finetune a 128B model, you should be generating your own datasets.
Not to speak for drummer but why can't you use what's available?
And why would you want to finetune this model further? If you have the raw resources to finetune a 128B model, you should be generating your own datasets.
As an ik_llama.cpp user I prefer the ik_llama.cpp specific IQ_K quants simply because they are more effcient and I want the best possible quality per GB.
As for further finetuning, getting an actually good dataset is far harder and more expensive then the raw resources to train a 128B, assuming we are converting the hours of research and iteration it takes to end up with a good dataset into money at least.
While the compute to finetuning a 128B is certainly not cheap it's probably not that impossibly expensive either (assuming we are talking about a relative lora finetune on <50M tokens here) relative to like hundreds of hours of work.
My point was more if you're at that point you should just do your own finetunes.
I also used to do finetunes on drummers work (cydonia) but realized over time id rather do my own as it was more satisfying.
The money spent on generating the dataset is not small but it's not outrageous either. Depends on the model you use. For example, you can do smaller models on runpod (up to 120B) and still come out ahead compared to API providers cost wise.
You do not need the most expensive, largest model for generating the dataset and I've proved that already in my latest tunes.
My point was more if you're at that point you should just do your own finetunes.
I also used to do finetunes on drummers work (cydonia) but realized over time id rather do my own as it was more satisfying.
The money spent on generating the dataset is not small but it's not outrageous either. Depends on the model you use. For example, you can do smaller models on runpod (up to 120B) and still come out ahead compared to API providers cost wise.
You do not need the most expensive, largest model for generating the dataset and I've proved that already in my latest tunes.
The hardest part of making a dataset is figuring out and leaning how to make on and finding out what does and doesn't work. The hardest most expensive (in terms of hours) part is figuring out what works and what dosn't.
Also how is syntetic data from a 120B going to help improve a 120B, isn't that just feeding a model it's own level of output making it worse? At minimum for that to work I imagine you'd need really good filtering potentially manually cherry picking samples good enough to train on.
Also don't address my second point about custom quants.
Also how is syntetic data from a 120B going to help improve a 120B, isn't that just feeding a model it's own level of output making it worse? At minimum for that to work I imagine you'd need really good filtering potentially manually cherry picking samples good enough to train on.
I had the same question. So I went and actually did it as an experiment. Turns out, it worked fine.
I generated data on Gemma 4 31B for Gemma 4 31B and to my absolute shock it changed how the model responded and people liked it.
So, take that as you will.