abhinavpgagi/personaplex-tool-calling

Tool calling for full-duplex speech-to-speech (PersonaPlex / Moshi)

Function calling on a full-duplex speech-to-speech model, without breaking the duplex property.

The model keeps listening and speaking while an external API call runs, and the result is spoken in the model's own voice and prosody — not spliced in as TTS, and without pausing the conversation.

Built on NVIDIA PersonaPlex (7B, itself built on Kyutai Moshi).

---

Why this is not obvious

A speech-to-speech model like Moshi has no text interface to hook a tool into. It emits, every 80 ms frame:

• audio tokens (what you hear), and
• one inner-monologue text token — the model's own running transcript of what it is about to say

There is no "assistant message" to append a tool result to, and no turn boundary to pause at. The two obvious approaches both break it:

approach	what breaks
Pause generation, call the API, resume	The model is full-duplex — it is always generating. Pausing produces dead air and desynchronises its sense of time.
Call the API, speak the result with TTS	Voice and prosody change mid-conversation. It stops sounding like one speaker.

What this does instead

Teacher-force the tool result into the inner monologue, one token per frame. The model then says it, in its own voice, as if it had thought of it:

 user audio ──(transcription)──┐
                               ├──> running transcript ──> LLM + tool schemas
 model's own \x02 text ────────┘                                  │
                                                        HTTP call (allowlisted)
                                                                  │
        ┌── inner monologue <── one forced token per 80 ms frame <─┘
        │
   the model speaks the answer itself — same voice, same prosody

Concretely, in forked_server.py's generation loop:

if pending_text_tokens:                    # a tool result is waiting
    forced_text = pending_text_tokens.popleft()   # override the model's own sample
elif _nudge["force"] > 0:                  # else a turn-taking nudge
    forced_text = _NUDGE_TOKEN
else:
    forced_text = None                     # else the model samples freely

tokens = lm_gen.step(codes[:, :, c:c+1], text_token=forced_text)

The result is drip-fed one token per frame rather than injected as a block, so the model's audio head generates speech for it at the natural rate. Generation never stops; the duplex property is preserved throughout.

A filler phrase ("let me check that") can be injected the same way the moment a call starts, so the pause while the API responds sounds deliberate.

Using it

The API list is supplied per session, base64-encoded in the X-Functions header on the WebSocket upgrade — so different callers get different tools without redeploying:

{
  "prompt": "Decide whether to call a function based on the live conversation.",
  "filler": "Sure, let me check that for you, one moment.",
  "allowed_hosts": ["api.open-meteo.com"],          // SSRF allowlist — required
  "functions": [{
    "name": "get_weather",
    "description": "Current weather for a location given latitude and longitude.",
    "endpoint": "https://api.open-meteo.com/v1/forecast",
    "method": "GET",
    "param_location": "query",
    "static_params": { "current_weather": true },
    "parameters": {                                  // OpenAI tool schema
      "type": "object",
      "properties": {
        "latitude":  { "type": "number" },
        "longitude": { "type": "number" }
      },
      "required": ["latitude", "longitude"]
    }
  }]
}

python client/personaplex_client.py \
  --url wss://<your-host>/api/chat \
  --api-key "$API_KEY" \
  --script my_persona.txt \
  --functions examples/weather.json \
  --input mic

Then ask it about the weather somewhere. It will say the filler, call the API, and speak the answer without ever stopping.

Layout

path	what
src/reasoner.py	transcript accumulation, LLM tool decision, result → injection
src/http_executor.py	declarative function spec → HTTP call; SSRF allowlist
src/session_config.py	per-session X-Functions config, validated
src/forked_server.py	PersonaPlex server + the injection bridge (see licence below)
client/personaplex_client.py	mic client; stereo recording; per-turn latency
bench/	turn-taking latency harness — see below
examples/weather.json	a working end-to-end example

Turn-taking latency harness

bench/voicebench.py measures the metric that matters for a voice agent:

latency = model_speech_start − caller_speech_end

Unlike VAD-based harnesses it synthesises the caller's turns, so speech-end is an exact sample rather than a VAD guess with unknown hangover bias. Onset is the first of three consecutive 20 ms windows above an RMS threshold, timestamped at the first window so sustain confirmation adds no upward bias. Validated against a mock server with a known injected delay.

It also reports what latency alone hides: dropped turns (the model stayed silent) and barge-ins (it talked over the caller) — both excluded from the latency figure rather than silently flattering it.

python bench/make_turns.py                       # synthesise caller turns (macOS `say`)
python bench/voicebench.py --url "$WS_URL" --runs 5

Requirements and caveats

• Gated weights. nvidia/personaplex-7b-v1 requires accepting the NVIDIA Open Model Licence. Weights are not included here.
• An LLM API key. The reasoner uses OpenAI for transcription and the tool decision (LLM_API_KEY). This is not self-contained speech-to-speech — the tool-decision loop is text and external.
• One conversation per process. The server holds a single asyncio.Lock around one LMGen. Multi-worker concurrency on a shared GPU was tested and collapsed — scale by process-per-GPU or by instance, not by workers per GPU.
• SSRF. The server makes HTTP requests to client-supplied URLs. allowed_hosts is enforced and private/loopback/link-local IPs are blocked — keep it set.
• No authentication upstream. PersonaPlex ships with none. S2S_API_KEY adds Authorization: Api-Key <key> checked before the WebSocket upgrade. Unset means the endpoint is open; the server logs a warning at startup.
• VRAM. ~19 GB in use on a 24 GB A10G. It has OOM'd under load; 24 GB is the practical floor and there is little headroom for long sessions.

Licence and provenance

• src/reasoner.py, src/http_executor.py, src/session_config.py, client/, bench/ — original work, © 2026 Abhinav Kalvacherla, Apache-2.0 (see LICENSE). Each file carries an SPDX header.
• src/forked_server.py — a fork of PersonaPlex's moshi.server, © NVIDIA CORPORATION and © Kyutai, MIT. The original notice is retained in the file. Modifications are listed in NOTICE.
• Model weights — NVIDIA Open Model Licence, not distributed here.