Usable context length?

#1
by belisarius - opened

Whats the maximum context this can take before producing nonsense?

The context window for Mixtral 4x7B is technically unlimited, but it operates with a 4K sliding window. Mixtral claims it's 32k but each expert was trained on 8k.
I run it with LMStudio set to 8k with the rolling context on, that and using Novelcrafter with custom prompts and the codex system I regularly get 16k or more but YMMV.

Sign up or log in to comment