Hello!
I’m inviting people to refract ai so we can do like collabs and stuff. @Hoglet-33 @Banaxi-Tech @ArushCodes @GGUFGuy ppl I pinged. I invited join if u want to like colab on the next T3!
Something is coming with NACR 3 Pro. So let's wait for that before T3
Something is coming with NACR 3 Pro. So let's wait for that before T3
Ok I am still up to T2.1 lol
sorry guys, im lost, whats T2/T3?
A arch i invented i dubbed it transformer 2 but it’s not official. Read thr paper I made in the org
oh okay
Make T3 special! Something like using Lion instead of AdamW? Or Mixture of value attention?, or something special
@ArushCodes i dunno… T2 is already pretty special but I’ll add more stuff slowly as it will like explode if I add too much unproven stuff. I think at that tiny size we don’t need MoD. Muon I could try but idk
@ArushCodes i dunno… T2 is already pretty special but I’ll add more stuff slowly as it will like explode if I add too much unproven stuff. I think at that tiny size we don’t need MoD. Muon I could try but idk
You can use Muon it gives good convergence, lion is a replacement for adamw if you want convergence and memory savings at a same time... NGRAM is a very safe to use thing, fasts convergence in almost no compute ... Mixture of heads gives more accuracy but I think as of the architecture T2 it's dangerous and volatile. My main account: https://huggingface.co/ArushBuilds here I am also building Pragya SLM from scratch . You can add my main account to organisation
hi guys