CalderaAI

/

30B-Lazarus

like 119

They are merging datasets, not models.
Any dataset can be trained against any base model. your GPT4xAlpaca dataset can train a 65b model as well as a 7b model.
They assembled these datasets and then trained a 33b model with it.

@teknium I suspect he used chansung/gpt4-alpaca-lora-30b it would not be possible to merge models of various sizes.
@ehartford This is merging lora's into models, and merging models directly, this is not a merging of datasets or training.

Part of the tools can be found here : https://github.com/ontocord/MDEL/tree/main/Model%20Merge%20And%20Analysis%20Tools

@Henk717 Why does the method (merging models directly) work? It is quite rare until recent months. Is there any paper/blog to describe this technology?

We're working on the paper, at the moment mostly research and experimentation by the team.