vessel-handoff-MiniCPM5-2B

A small, fast decision model for voice assistants: does this spoken request need the agent's tools, or can the voice just answer? Fine-tuned from openbmb/MiniCPM5-2B for Hermes Vessel's private Local voice. There, a full-duplex speech model (MiniCPM-o 4.5) does the talking but can't use tools, Whisper turns the user's speech into text, and this model decides, in about a tenth of a second on a GPU or 1-2 s on a CPU, when to hand the user's words to the Hermes agent.

File	What	Size
MiniCPM5-2B-vessel-handoff-Q6_K.gguf	recommended (used by Vessel's installer; same scores as Q8_0)	2.1 GB
MiniCPM5-2B-vessel-handoff-Q8_0.gguf	near-lossless	2.7 GB
lora/	the LoRA adapter (PEFT) on openbmb/MiniCPM5-2B	100 MB
router_prompt.json	the system prompt, tool definition and request settings it was trained and measured with

Use

llama-server -m MiniCPM5-2B-vessel-handoff-Q6_K.gguf --jinja -c 4096 --port 8799   # llama.cpp b12177 or newer

Send the system prompt and use_tools tool from router_prompt.json, the user's transcribed words as the user message, temperature: 0, thinking off (chat_template_kwargs: {"enable_thinking": false}). Only the decision is needed, so max_tokens: 4 is enough: a reply that starts with <function means hand off (pass the user's own words to the agent); anything else means chat (let the voice answer). If a transcript holds two sentences and the whole reads as chat, also check the last sentence alone.

Why fine-tune

The base model with thinking off is fast but refused instead of calling tools (4 of 57 held-out tool requests called). With thinking on it called tools but took 0.6 s and rewrote requests, sometimes into a different task ("remind me to call Sam" became "check memory for any information about Sam"). The fine-tune does the right thing with thinking off, and passes the user's words on unchanged.

Results

Test (written transcripts)	Base, thinking off	Base, thinking on	This model
Held-out tool requests handed off	4/57	54/57	57/57
Held-out chat kept as chat	40/40	40/40	40/40
Hand-offs with words the model added	-	52/54	0/57
Fresh phrasings, not from any training template	14/24	-	24/24
16-sentence benchmark	10/16	15/16	16/16

Quantized: Q8_0 and Q6_K kept every score above (also in decision-only mode, 4 tokens, on CPU: 57/57, 40/40, 24/24); a Q4_K_M build lost one chat and one fresh case, so it isn't published. Decision time: ~0.1-0.2 s on an RTX 5080, ~1.5-2 s on 8 laptop efficiency cores (Core Ultra 5 125H) in decision-only mode.

Measured on written transcripts and recorded speech in one TTS voice; not yet on a wide range of real voices.

Training

LoRA r=16, alpha=32, on all attention and MLP projections; 3 epochs; ~6 minutes on one RTX 3090 (11.6 GB peak). About 560 spoken-style examples:

• Tool requests from slot-filled templates: weather, the user's machines and services, calendar, smart home, personal memory, the web, files and messages, with transcript noise (fillers, lower case, no punctuation). The target is a use_tools call whose request is the user's own words, cleaned, never expanded.
• Chat: small talk, general knowledge, opinions, feelings, and hard negatives ("what's a parlay?" vs "what did Alex and I bet on?"). Replies come from the base model's own thinking-mode answers, so training changes only when it calls a tool, not how it talks.
• Held out for evaluation: whole slot values (a city, a host, a person, a store, a league) and the benchmark sentences.

The training set is synthetic but written around one person's life (names, places, machines), so it isn't published; the recipe and scripts are described in the Vessel repository (docs/local-voice.md).

Limits

• English only. Built for one agent's tool set (files, web, memory, calendar, smart home, code); other agents may need different examples.
• It decides; it doesn't answer. Chat replies it writes are not meant to be spoken (the voice model does that).

License

Apache-2.0, like the base model openbmb/MiniCPM5-2B.