What is the max_new_tokens of model "Mistral-7B-Instruct-v0.1-GGUF"?
#5
by
manuth
- opened
hello ! everyone ^^
can I have your suggestion on the max_new_tokens of model "Mistral-7B-Instruct-v0.1-GGUF"?
thx for sharing in advanced
Hmm not sure if llama cpp has sliding windows attention yet.
If it doesn’t, I think it should be 8k but if it does it should be 32k