Model is failing to load in text web UI or Koboldcpp

#2
by jlopez-dl - opened

dolphin-2.9.1-qwen-110b-Q4_K_S.gguf

I tried concatenating both files and leaving them the way they are, in a folder.
The error is:
KeyError: 'tokenizer.ggml.bos_token_id'
And when I try to souround the bos in a try catch like this:

        try:
            bos_token = metadata['tokenizer.ggml.tokens'][metadata['tokenizer.ggml.bos_token_id']]
        except KeyError:
            bos_token = eos_token

Any ideas? I tried with Q4_K_S and Q4_K_M

hmm odd, nothing off the top of my head.. I'll try to look into it, maybe needs a metadata update (would be tragic for bandwidth

@jlopez-dl

Can you make sure you're fully updated? I just tried it with latest llama.cpp directly and it worked fine

I created a super-simple script:

split_file1="/workspace/dolphin-2.9.1-qwen-110b-Q4_K_S.gguf/dolphin-2.9.1-qwen-110b-Q4_K_S-00001-of-00002.gguf"
split_file2="/workspace/dolphin-2.9.1-qwen-110b-Q4_K_S.gguf/dolphin-2.9.1-qwen-110b-Q4_K_S-00002-of-00002.gguf"
merged_file="/workspace/dolphin-2.9.1-qwen-110b-Q4_K_S_1.gguf"
cat "$split_file1" "$split_file2" > "$merged_file"
if [ $? -eq 0 ]; then
echo "Files successfully merged into $merged_file"
else
echo "An error occurred during the merge process"
fi

I suppose this is correct?.

I'll try with llama.cpp not text_web_ui and check.
Thanks!

Ooo no, you can't cat files together anymore like that. You either need to just point at the first part or compile and use the gguf-split tool

Sign up or log in to comment