Will this work at 4K or 8K context in oobabooga yet?
#2
by
Goldenblood56
- opened
Will this work at 4K or 8K context in oobabooga yet? I noticed Oobabooga seems to now support 70B GGML at up to 4K is that right? With the Llama 2? Does this update apply to using models like this at either 4K or 8K? Just wondering.
And if so does anyone know optimal settings in Oobabooga?
For
n_gqa and rms_norm_eps?
I know how to do the rest of the settings including "compress_pos_emb" but the above two I'm not sure about for 33B. I know what to set them for 70B llama 2.
I feel like this model is not supported yet but maybe it is?