PrunaAI

/

Phi-3-mini-128k-instruct-GGUF-Imatrix-smashed

like 60

Using koboldcpp 1.63, and as the title says. This also happens with 4k version

Are you exclusively getting this in koboldcpp or also when you use llama.cpp itself ?

the latest llama.cpp version support phi 3 now, so this should be closed.

Great!

This was converted before Phi 3 support was merged in llama.cpp so I reopened it.
Since the script convert-hf-to-gguf.py was also changed to support the new phi 3 architecture it, the gguf is outdated; might be using wrong tokens or are missing, it might be worth redoing the gguf in this repo

a new phi 3 mini 128k gguf a day ago shows;

general.architecture	phi3
general.name	Phi3

This one has

general.architecture	llama
general.name	phi3

Was this converted before Phi 3 support was merged in llama.cpp?
since the script convert-hf-to-gguf.py was also changed to support it, the gguf might be
using wrong tokens or are missing

We converted it with a custom PR on llama.cpp but we are in the process of updating the files with the latest version main. I will ping here when the update is done :)

https://github.com/ggerganov/llama.cpp/issues/6849#issuecomment-2074899603
It seems that 128K context phi-3 is still being worked on, so it might be better to focus on the phi-3 4K instead.

https://github.com/ggerganov/llama.cpp/issues/6849#issuecomment-2074899603
It seems that 128K context phi-3 is still being worked on, so it might be better to focus on the phi-3 4K instead.

Ah okay thanks for the heads up, will do! :)

@Bakanayatsu 4k quants are now updated here