merge
This is a merge of pre-trained language models created using mergekit.
Merge Details
Merge Method
As we know that Fine tuning Only updates the final layer , as well as extration and derankng with lord also extracts this last layer! / Penultimate layer: Hence when fine tuning models ; you CANNOT fine tune on TOP of the fine tuning;
Hence merging!
So collecting finetuned models and mmerging retains the skills learned by both models wherre as finetuning on top of fine tuning replaces the final layer... even applying loras on top of loras resets you!
Hence Finetune!,MERGE!..... Rinse and repeat! Upgrading! Or you can reload the same lora for furthr fine tuning, as some loras even become ery large due to the number of epochs! Essentially a single layer highly tuned expert!!
So the next projext is the Mixture of Adapters !.... MoMerge! PhatGoose etc: creating an experts model from loras ! (hopefully 32 models to create a frankenmerger to be directly merged into the main model and re-alligned in!)
WATCH THERSE SPACES !
Models Merged
The following models were included in the merge:
- Y_Chroma <<<<<<<<<<<< 6 models merged (chat comercial based models, ie: zephr, openchat, antropic etc)
- LeroyDyer/Mixtral_AI_CyberTron_Ultra <<< Model being Upgraded (remixed with CyberBoss/SmartBrain/CyberCoder)
- X_Chroma <<<<<<<<<<<< 6 model Merged (maths Focused from wizardMath to MetaMath)
- Downloads last month
- 302