MTP GGUF?
Can I just convert this to GGUF and use it in llama.cpp as usual? Any potential pitfalls I should be aware of?
Thanks
@xms991 Not as a drop-in conversion that we can currently recommend. This repo contains a separate draft head, not a complete model. Converting it to GGUF alone doesn't establish that llama.cpp will load and use it for MTP; the converter and runtime need to support this architecture's draft-head tensors and speculative path.
We haven't validated this head with llama.cpp/GGUF. The SGLang runtime patches here are also not portable to llama.cpp. If you want to try it, which llama.cpp version and base GGUF are you using? The first checks would be that conversion preserves the head tensors, drafting is actually active, and greedy outputs match the same base with MTP off before measuring acceptance or speed.
Kindly support to provide GGUF. llama.cpp can load MTP only GGUF, even if you do not provide merged weights.
@engrtipusultan Thanks for clarifying! We’ve converted a standalone MTP GGUF; it still needs to pass our llama.cpp correctness checks before release.