NousResearch

/

Yarn-Llama-2-13b-128k

like 113

CodeLlama 13B or 34B YaRN 128k would be a very potent model.

I mean CodeLlama is already 100k, not sure why you need that extra 28k.

I've tested Code Llama up to 16k with vLLM. Which LLM engine to use to run it at 100k?

I've tested Code Llama up to 16k with vLLM. Which LLM engine to use to run it at 100k?

Most LLM inference tools/applications support RoPE(Which is what CodeLlama uses). It's just that it uses way to much RAM/VRAM(So do YaRN). I'm not sure if you can run it.