So this isn't SOLAR?
Borealis-10.7B is a 10.7B model made of 48 Mistral 7B layers, finetuned for +70h on 2xA6000 on a big RP and Conversational dataset with llama2 configuration of Axolotl, like SOLAR.
Shouldn't it be better to just finetune base SOLAR? I mean it's not just a normal Mistral frankenmerge, it's also further pretrained with 3T tokens. While this one have the same concept but with much fewer training data?
Borealis-10.7B is a 10.7B model made of 48 Mistral 7B layers, finetuned for +70h on 2xA6000 on a big RP and Conversational dataset with llama2 configuration of Axolotl, like SOLAR.
Shouldn't it be better to just finetune base SOLAR? I mean it's not just a normal Mistral frankenmerge, it's also further pretrained with 3T tokens. While this one have the same concept but with much fewer training data?
Yeah no, it's not solar. Undi is doing experiments here, wanted to see what the model would be like primarily trained on RP data.