Can we help fund 30b?
Hi is there a way to help support this for a 30b version? Happy to chip in. 30b for me is where Llama starts to be truly coherent and vast, but if computing cost is a problem I'm happy to help fund.
Hey! We do plan on scaling up, but there are no plans to accept external funding - at least for now. We're already in contact with some people who might be able to provide the compute resources necessary. If the situation changes, we'll let everyone know via an announcement.
heck yea
What model would be used for a 30B that have a compatible (fully open source) license?
Huh? Llama 7b was used for this.... llama 30b would be used, it's just a B parameter change
Llama 30B uses what license? Is it able to be used for commercial applications, including ones with x-rated outputs?
Go Google it, it's the same license for all llama.
Will the 30b version please end in PT or safetensor so I can use it in oobabooga? Pretty please?
Would you consider using Vicuna-13b to achieve better training results?
Just making clear to anyone reading that I am still not accepting money.
@Deltaadams if you want to advertise your Gofundme, please be a little clearer about the fact that you're training your own model - the way you posted was pretty confusing and made it seem like we were behind the fundraiser ourselves.
Also, to answer some questions:
What model would be used for a 30B that have a compatible (fully open source) license?
@hrabanazviking The plan for now is LLaMA, then after the initial release, I'll wait for a decent 30B foundational model (RedPajama? OpenLLaMA?) with an open license and retrain so I can release with a more permissive license, since I'm not a fan of LLaMA's.
Will the 30b version please end in PT or safetensor so I can use it in oobabooga? Pretty please?
@Humeee33 Unfortunately, no can do as long as the model based on LLaMA. I have to distribute XORs because of the license. Hopefully RedPajama or OpenLLaMA pull through and I can train a version on top of those, and then I can release normal PyTorch files.
Would you consider using Vicuna-13b to achieve better training results?
@Jackdiy I don't think it'd be a great move. Vicuna is trained exclusively for ChatGPT-like chats as far as I understand, and the use-case for this model is more entertainment/fictional characters. Our training data already includes some of Vicuna's data, but in very moderate amounts as to not bias the model towards sterile ChatGPT-like responses.