16K or 32K context length?
#4
by
ddh0
- opened
Hello, does this model only support 16K, or can it be extended to 32K? Since the base model is 32K context length, I am confused why this one is 16K.
Thank you!
Hi ddh0,
Yes, you can extend the 16k chat model to 32k simply by changing max position embeddings in both config and during inference. But expect the response quality to degrade.
thanks,
Nuo
Cool, thank you for the clarification!
ddh0
changed discussion status to
closed