Thanks. Did you use all 3 end tokens?
Although I can't test these GGUFs yet they're almost certainly correctly made. The two apps I use need to be updated, and GPT4ALL even says model phi3 not supported when it fails to load. So I downloaded and will test later.
In the mean time I have a quick question, did you use all 3 end tokens (pasted below)?
I ask because every other phi3 instruct gguf I tested, including Microsoft's official Q4 & Q8 versions, have a bug were they often talk past the end token and display formatting and irrelevant behind the scenes information. And according to Microsoft this is primarily due to the use of 3 different end tokens, which was initially omitted from generation_config.json.
"eos_token_id": [
32000,
32001,
32007
],
Hmm I didn't manually change it so if it's not by default this doesn't have it either
However, I did test using llama.cpp main branch and the ./main after building and generation stops perfectly
@bartowski Thanks, then it's probably working right. I'm looking forward to testing a Phi3 Instruct that reliably stops when it should. So far not one has.