"Tokenizer config may be incorrect" - also, unused tensors?
0.04.535.876 W load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
0.04.596.684 W model has unused tensor blk.41.attn_norm.weight (size = 10240 bytes) -- ignoring
0.04.596.688 W model has unused tensor blk.41.attn_q.weight (size = 16711680 bytes) -- ignoring
0.04.596.691 W model has unused tensor blk.41.attn_kv_a_mqa.weight (size = 1566720 bytes) -- ignoring
0.04.596.693 W model has unused tensor blk.41.attn_kv_a_norm.weight (size = 2048 bytes) -- ignoring
0.04.596.696 W model has unused tensor blk.41.attn_k_b.weight (size = 2228224 bytes) -- ignoring
0.04.596.698 W model has unused tensor blk.41.attn_v_b.weight (size = 2228224 bytes) -- ignoring
0.04.596.700 W model has unused tensor blk.41.attn_gate.weight (size = 87040 bytes) -- ignoring
0.04.596.703 W model has unused tensor blk.41.attn_output.weight (size = 11141120 bytes) -- ignoring
0.04.596.705 W model has unused tensor blk.41.ffn_norm.weight (size = 10240 bytes) -- ignoring
0.04.596.708 W model has unused tensor blk.41.ffn_gate_inp.weight (size = 5242880 bytes) -- ignoring
0.04.596.710 W model has unused tensor blk.41.exp_probs_b.bias (size = 2048 bytes) -- ignoring
0.04.596.712 W model has unused tensor blk.41.ffn_gate_exps.weight (size = 432537600 bytes) -- ignoring
0.04.596.715 W model has unused tensor blk.41.ffn_up_exps.weight (size = 432537600 bytes) -- ignoring
0.04.596.717 W model has unused tensor blk.41.ffn_down_exps.weight (size = 534773760 bytes) -- ignoring
0.04.596.719 W model has unused tensor blk.41.ffn_gate_shexp.weight (size = 2088960 bytes) -- ignoring
0.04.596.722 W model has unused tensor blk.41.ffn_up_shexp.weight (size = 2088960 bytes) -- ignoring
0.04.596.724 W model has unused tensor blk.41.ffn_down_shexp.weight (size = 2088960 bytes) -- ignoring
Llama.cpp outputs the above, I'm curious if it's something I forgot to do or change? I'm running with llama-server -m Ling-3.0-flash-AD-IQ3_XXS-00001-of-00002.gguf -ngl -1 -c 60000 -np 1 -ctk q8_0 -ctv q8_0 --chat-template-file ling-3.0-flash-chat-template.jinja --load-mode none --reasoning-preserve
and it has issues like writing some_file instead of some-file and sometimes puts in chinese characters. Ideas?
Edit: I should add that im using llama.cpp-pr26608
Hi @je0923 !
The problem is entirely on llama.cpp-pr26608's side.
I've recently fixed a bunch of bugs in our llama.cpp turboquant fork and currently doing a release into master, so I'd highly suggest you using it:
https://github.com/AtomicBot-ai/atomic-llama-cpp-turboquant
Please build it yourself for now using cmake. I'm going to publish a release, but it's going to take ~2 hours from now. Sorry(
Also i'm updating quants right now to fix a recently found bug, stay tuned!
Here I checked again IQ2_XS quant quickly for you to see if it's ok:
As you can see, we have exactly 42 layers, so our build runs all layers. Your PR build substracts one from the other and lands on 41.
Ling 3.0 flash has 43 layers - 42 + MTP, which I currently don't support.
Also, I suggest dropping -ctk q8_0 -ctv q8_0. On MLA the cache holds an already low-rank compressed latent, and quantizing that is a lot more lossy than quantizing a normal KV cache. We haven't validated it on this architecture.
Edit: Wait, did you update the quants themselves? Do I have to redownload? I haven't yet, if so ignore the below haha
Edit 2: Oh, I see someone mentioned the same bug! Cool, I'll wait for that fix then :D Thanks so much!
https://github.com/ggml-org/llama.cpp/pull/26608#issuecomment-5207876317
Hmm, the tensor thing is fixed - there's no warnings about that - but I may be doing something wrong with the tokenizer. I still get 0.04.520.263 W load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
I'm using the chat template at https://huggingface.co/inclusionAI/Ling-3.0-flash/raw/main/chat_template.jinja
When I run the model and use it with pi, it ends up doing tool calls and literally putting the arguments within the name of the file:
The user wants me to create a new file with the alphabet. Simple enough. I'll create a file containing the alphabet (a-z).
write ~/Desktop/model_runner/alphabet.txt</arg_value><arg_key>content</arg_key>
<arg_value>abcdefghijklmnopqrstuvwxyz
I feel like there's probably a super simple fix haha. I'm in pi with the llama cpp built from the link you gave and i use normal kv cache. Any ideas? It's OK if this is just a me thing, but I was just curious 😃
About a tool call corruption - isn't not the quants, reproduced it on bloomer010's files too. It's llama.cpp tool call parsers fault. It infers arg_value terminator from a sample render, picks up \n and the model doesn't emit that newline between arguments
This fix has to be in the parser. And also - single-argument tools parse fine 😄