QuantFactory

/

Meta-Llama-3-8B-Instruct-GGUF

like 250

llm_load_vocab:
llm_load_vocab: ************************************
llm_load_vocab: GENERATION QUALITY WILL BE DEGRADED!
llm_load_vocab: CONSIDER REGENERATING THE MODEL
llm_load_vocab: ************************************
llm_load_vocab:

Hi there!

I've tried some quantized versions of this model and ran into an issue. I use llama-cpp-python for model inference. When I provide a question, I get infinite random characters as the result (see screenshot). But when I create a local model from the same quantized gguf by using Modelfile for Ollama inference, then everything works fine. So the issue is that Ollama works, and llama-cpp-python provides random output. The same behavior was noticed with a couple other models, like defog/llama-3-sqlcoder-8b.

Is anyone here experiencing same issues?

@liashchynskyi Yes! I'm having the same issue with defog/llama-3-sqlcoder-8b. I'm using LangChain with llama-cpp-python - only GGUF models. I'm looking to use GGUF files others have created - I can look into generating my own if that's the only solution.

Output from defog/llama-3-sqlcoder-8b:

@jaycann2 can you try QuantFactory/Meta-Llama-3-8B-Instruct-GGUF-v2
I'll update the defog quants today if you are facing issues with them

@munish0838 But why do we receive random outputs? I've tried to quantize the original model myself and ran into the same issue.

@jaycann2 can you try QuantFactory/Meta-Llama-3-8B-Instruct-GGUF-v2
I'll update the defog quants today if you are facing issues with them

Thanks @munish0838 - I tried yesterday and got the same result. I'd be interested to see if you are able to duplicate the issue on you end, with the GGUF version. If not, I might be able to learn what's going on from your code.

@jaycann2 I updated the quants yesterday in this repo and defog-sql-llama repo, they are both working perfectly for me