Help

#4
by damerajee - opened

So I got inspired by you and I wanted to do something similar to this just for experimenting with it ,so I implemented Llamoe which is this model https://huggingface.co/damerajee/Llamoe-test , and I keep getting this error
RuntimeError: The size of tensor a (200) must match the size of tensor b (2048) at non-singleton dimension 3 can you fix this
So I look through your modeling code and it seems as if most of it is like similar to the Llama architecture ,so I copied mostly from your code but started changing stuff here and there , I really tried fixing the error , but this is the first time I'm coding a MOE model so any help please

Sign up or log in to comment