Any chance of preserving the vision capabilities (for quantization)
#15
by AtomicGrimdark - opened
Trying to quantize this model as a test, it seems that llama.cpp's convert_hf_to_gguf.py script cannot extract the multimodal parts (mmproj) - runs into ValueError: Can not map tensor 'lm_head.weight'
I think it's happening while the gguf script is indexing the tensors. Which is interesting, because I can get it to work with an abliteration of gemma 4. Probably a mismatch of expected tensor dimensions of the lm_head.. or something in that general area, but it's way out of my understanding at the moment.
If it's on purpose it's fine, I mean, text generation is the focus here 😃
If anyone has success on getting vision to work, drop me a line here.