The model isn't currently supported by LlamaCPP, which is what KoboldCPP uses. Typically, you can tell if it'll work or not if in the Files tab you see stuff like modeling_model.py, configuration_model.py, ext.

As a potential solution, I recommend making a feature request on the LlamaCPP GitHub page to see if the developers or community members might be able to implement support for this model. If the request is accepted and implemented, there's a good chance that the model will be usable in future versions of KoboldCPP.

i thought GGUF was the format for LlamaCPP? and this is a projector, you use it alongside Llama 3, which absolutely is supported by LlamaCPP?

unless I've gotten one or both of these things wrong?

@baphomet-eight you're correct, sorry for the late reply I don't check hf notifications often enough.

The base model can be any llama3 base model afaik though I've only really tested with llama3 7b and to load the vision projector:

Python Usage

from llama_cpp.llama_chat_formats import Llama3VisionAlpha
# ... same usage as other multimodal formats see https://llama-cpp-python.readthedocs.io/en/latest/#multi-modal-models

Server usage

Set chat_format to llama-3-vision-alpha