Model is failing to load in text web UI or Koboldcpp
dolphin-2.9.1-qwen-110b-Q4_K_S.gguf
I tried concatenating both files and leaving them the way they are, in a folder.
The error is:
KeyError: 'tokenizer.ggml.bos_token_id'
And when I try to souround the bos in a try catch like this:
try:
bos_token = metadata['tokenizer.ggml.tokens'][metadata['tokenizer.ggml.bos_token_id']]
except KeyError:
bos_token = eos_token
Any ideas? I tried with Q4_K_S and Q4_K_M
hmm odd, nothing off the top of my head.. I'll try to look into it, maybe needs a metadata update (would be tragic for bandwidth
Can you make sure you're fully updated? I just tried it with latest llama.cpp directly and it worked fine
I created a super-simple script:
split_file1="/workspace/dolphin-2.9.1-qwen-110b-Q4_K_S.gguf/dolphin-2.9.1-qwen-110b-Q4_K_S-00001-of-00002.gguf"
split_file2="/workspace/dolphin-2.9.1-qwen-110b-Q4_K_S.gguf/dolphin-2.9.1-qwen-110b-Q4_K_S-00002-of-00002.gguf"
merged_file="/workspace/dolphin-2.9.1-qwen-110b-Q4_K_S_1.gguf"
cat "$split_file1" "$split_file2" > "$merged_file"
if [ $? -eq 0 ]; then
echo "Files successfully merged into $merged_file"
else
echo "An error occurred during the merge process"
fi
I suppose this is correct?.
I'll try with llama.cpp not text_web_ui and check.
Thanks!
Ooo no, you can't cat files together anymore like that. You either need to just point at the first part or compile and use the gguf-split tool