MTP for MoE?
#1
by jacek2024 - opened
So MTP works on your setup for MoE? According to llama.cpp implementation it helps only for Dense
Yeah, the general state of MTP support is rather sketchy right now but I ran a test using the raw Transformers model and that worked really well. It's a bonus fact, not a definitive statement. I don't do those. ;-)
So I tested the MTP inside Llama.cpp just now since it got merged but yeah, performance is far, far worse. So no more bonus fact!
Gryphe changed discussion status to closed