What is 256k?
I was hoping that it was context size, but that is only 8K, so what is this number?
It's 256k
Sorry I forgot to say please.
Is it the number of examples from dolphin dataset on which llama 3 was trained?
It's 256k context size. The model can process text with a lenght up to 256k context. Hope this answers your question.
EDIT: I think you have to adjust the rope tetha and similar stuff to get the 256k to work.
Damn, 256K! That's twice than what GPT 3.5 has. How much RAM does this require?
Damn, 256K! That's twice than what GPT 3.5 has. How much RAM does this require?
That is good question, however, I never got past 4K due to limited RAM.
I get that you are replying to him asking about the context size but it just makes it seem like you werent really answering the question based on how he asked
Ask Google/chatgpt/whatever "how do I use RoPE to set 256k context in (ollama, ooba, vLLM, tgi, etc)"
Damn, 256K! That's twice than what GPT 3.5 has. How much RAM does this require?
That is good question, however, I never got past 4K due to limited RAM.
How much RAM do you have?
Also rombodawg is right, the owner didn't answer the OP's question. If it wasn't for CyberTimon, I wouldn't be sure that was really the context size either.
People need to be straightforward when talking about boring shit like tech. Trying to be funny in this kind of environment is just cringe.
If it's 256k context length, why is "max_position_embeddings": 8192?
I will update the max_position_embeddings setting
Fixed
By the way 1m is uploading.
https://huggingface.co/cognitivecomputations/dolphin-2.9-llama3-8b-1m
Fixed
Where does the "max_position_embeddings": 524288 come from if it's 256k?
Thats a good fucking question lol