How to use it?
I tried loading it with koboldcpp but it asks me for a model in the field "Model:" when I click "launch"
And if I select the same model I get the following error:
I tried loading it with oobabooga with the following arguments:
--model Kukul-Stanta-0.420-32k-7B-0.2-Q4_K_M-imat.gguf --loader llama.cpp --n-gpu-layers 23 --multimodal-pipeline llava-7b
But it doesn't have any vision capabilities
From the model card README.md:
If you want to use vision functionality:
- Make sure you are using the latest version of KoboldCpp.
To use the multimodal capabilities of this model, such as vision, you also need to load the specified mmproj file, you can get it here or as uploaded in the repository.
- You can load the mmproj by using the corresponding section in the interface:
- For CLI users, you can load the mmproj file by adding the respective flag to your usual command:
--mmproj your-mmproj-file.gguf
Perhaps @Nitral-AI can comment on the works/doesn't on Ooba, as I don't use it. GGUF models will perform better in KoboldCpp.
What you need to load in the LLAVA field in KCPP is the MMPROJ file, as explained in the card page, you can get it here, or inside the mmproj folder in this repo.
You load the model itself where it says model. You can find the LLaVa mmproj file in any of the various vision repo's.
Thank you, I didn't read the instructions correctly