OpenVINO IR model with int8 quantization

#2
by fakezeta - opened

Hi, I've converted you model to OpenVINO IR format.
Template to use it on LocalAI are added to the README.md file.

NousResearch org

is there instructions on how to configure function calling for Heremes-2-Pro with LocalAI?

Support for function calling on transformers Is in this PR with example instruction.

I still haven't had the chance to test it: will report here as soon as possible.

Sorry for the late reply: function calling is working both for Hermes-2-Pro and Hermes-2-Theta.
I tested it with AnythingLLM agents.

The configuration needed has been added to the README.md

NousResearch org

glad to hear that -- could you open a PR with an example notebook on Hermes-Function-Calling repo?
https://github.com/NousResearch/Hermes-Function-Calling/tree/main/examples

I'll do my best: I'm not a developer just a passionate guy.

Sign up or log in to comment