Failed to use Q8 model in LM Studio
#1
by
rorrerror
- opened
Hi. I download and load the Q8 model in LM Studio 0.2.18 on M2 MAX MAC with 64GB RAM. However, the model cannot generate meaningful responses. Any idea? Thanks for any suggestions.
@rorrerror that's curious, can you try a smaller size to verify? I just tried with Q2 and it produced coherent results
The Q8 model In llama-cpp-python running on my M2 Max 64GB gave me same result when I set context length > 8000.
length < 8000 gives me normal outputs.
Ah interesting, the model only supports 8000 ctx so that might be why