Kupe-ThinkSpark-Realtime-270M

A realtime conversation referee for voice agents.
Decides when to listen, when to hold, when to stop talking — in ~3 ms, on CPU or GPU.

Research · Docs · Kupe

---

What it does

Voice agents fail at the seams: they talk over people, they wait too long after someone finishes, they keep talking when interrupted. ThinkSpark-Realtime is a small model that sits beside the call and answers one question every 80 ms — who has the floor right now, and what should the agent do about it?

It emits one control flag per audio frame:

flag	meaning
LISTEN	user has the floor, stay quiet
HOLD	user paused mid-thought, do not jump in
INCOMPLETE	utterance is unfinished
TURN_END	user is done — commit and reply
BARGE_SOFT	user started over the agent — duck audio
BARGE_HARD	user is actively interrupting — stop speaking now
CONTINUE	agent may keep speaking
PREFETCH_LLM	safe to start a speculative LLM call
COMMIT_LLM	play the prefetched reply
SILENCE_BREAK	dead air — say something

It is not an ASR model, not an LLM, and not a TTS. It is the floor-control layer between them.

Languages

English, Hindi, and Gujarati only. The audio understanding was built on en/hi/gu speech; behaviour on other languages is untested and not supported.

Performance

metric	result	target
VAD F1	0.976	≥ 0.85
Barge-in F1	0.861	≥ 0.85
False-barge rate	0.000	≤ 0.05
Control macro-F1 (±240 ms)	0.860	—
Decode latency p50 / p95	2.9 / 4.2 ms	p95 ≤ 40 ms

Latency measured per frame on a single GPU; CPU inference stays well inside the realtime budget. Frames are 80 ms, so the model is roughly 20× faster than the audio it consumes.

Install

pip install "kupe[thinkspark]"

Pulls in torch, transformers, numpy, and huggingface_hub. Add sounddevice for microphone capture. Node/TypeScript: npm install kupe-sdk.

Quickstart

Audio in, decisions out — from a mic, a call leg, a file, anything. The weights download from this repo automatically on first use.

from kupe import ThinkSpark

ts = ThinkSpark()                       # device="auto" -> cuda | mps | cpu

for decision in ts.stream(source="mic"):
    print(decision.flag, decision.latency_ms)

Your own audio instead of the mic — any iterable of float32 numpy arrays:

for decision in ts.stream(source=my_audio_frames, sample_rate=24_000):
    print(decision.flag, decision.spoken, decision.latency_ms)

Barge-in only means something while the agent is talking, so tell the model what the agent is doing:

ts.set_context(agent_text="Your balance is four thousand two hundred rupees")

for decision in ts.stream(source="mic", agent_state="TTS_SPEAKING"):
    if decision.flag == "BARGE_HARD":
        tts.stop()
    elif decision.flag == "TURN_END":
        llm.commit()

agent_state is one of IDLE, LLM_GEN, TTS_SPEAKING, TTS_DONE.

TypeScript

import { ThinkSpark } from "kupe-sdk";

const ts = new ThinkSpark();

for await (const decision of ts.stream("mic")) {
  console.log(decision.flag, decision.latencyMs);
}

Device selection

ts = ThinkSpark(device="cuda")   # "cuda" | "mps" | "cpu" | "auto" (default)

Loading it yourself with transformers

kupe wraps a plain PyTorch module built on a transformers backbone. If you would rather wire the pieces up yourself:

import torch
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer

from kupe._thinkspark import vocab
from kupe._thinkspark.model import ThinkSparkModel
from kupe._thinkspark.mimi_codec import MimiEncoder
from kupe._thinkspark.inference import StreamingReferee, FrameInput

BASE = "google/gemma-3-270m"
REPO = "anuj-inavlabs/Kupe-ThinkSpark-Realtime-270M"

tok = AutoTokenizer.from_pretrained(BASE)
tok.add_special_tokens({"additional_special_tokens": vocab.ALL_SPECIAL_TOKENS})

encoder = MimiEncoder(device="cpu")            # kyutai/mimi, 24 kHz -> 80 ms frames

model = ThinkSparkModel(base_model=BASE, codebook_size=encoder.codebook_size,
                        gradient_checkpointing=False)
model.resize_token_embeddings(len(tok))
model.load_state_dict(torch.load(hf_hub_download(REPO, "model.pt")), strict=False)

referee = StreamingReferee(model.eval(), tok, system_prompt="...", device="cpu")

enc = encoder.encode_waveform(waveform, 24_000)
for i in range(enc.num_frames):
    r = referee.step(FrameInput(cb0=int(enc.cb0[i]), energy=float(enc.energy[i]),
                                f0=float(enc.f0[i]), agent_state="IDLE"))
    print(r.flag, r.decode_ms)

Why not AutoModel.from_pretrained alone?

transformers gives you the Gemma-3-270M backbone, but this model is not a plain causal LM — it has a multi-modal front end (Mimi audio-token embeddings + prosody projection + segment/state embeddings) and three output heads (control flags, VAP, spoken text) that live outside the transformers class hierarchy. It also needs the Mimi codec in front of it to turn a waveform into the audio tokens it reads.

transformers alone cannot express that graph, which is why the weights ship as a model.pt state dict plus the thin ThinkSparkModel wrapper above. Everything under kupe._thinkspark is plain PyTorch — no custom CUDA, no compiled ops, no trust_remote_code.

Realtime demo

examples/mic_demo.py listens on your microphone and prints a live, colour-coded terminal view of every decision — flag, latency, and a running Mimi-encode / model-decode / headroom breakdown against the 80 ms frame budget.

pip install "kupe[thinkspark]" sounddevice rich
python examples/mic_demo.py

Kupe-ThinkSpark-Realtime-270M — live session
┌──────────┬──────────────────┬─────────┬───────────┐
│ time     │ flag             │ spoken  │ decode ms │
├──────────┼──────────────────┼─────────┼───────────┤
│ 09:41:03 │ LISTEN           │         │      2.87 │
│ 09:41:04 │ BARGE_HARD       │         │      3.12 │
│ 09:41:05 │ TURN_END         │         │      2.94 │
└──────────┴──────────────────┴─────────┴───────────┘
frames   1284   mimi  11.3 ms   model  2.98 ms/frame   budget used  3.7%   headroom  77.0 ms

Intended use

Built for realtime voice agents: phone support, in-app voice, IVR replacement, live assistants — anywhere an agent and a person share one audio channel and have to take turns without stepping on each other.

Limitations

• en / hi / gu only. Other languages are out of scope.
• CANCEL_LLM is not supported. The flag exists in the label space but the model does not emit it; do not build cancellation logic on it.
• Expects 24 kHz mono audio. Resample before calling.
• Designed for two-party conversation. Multi-speaker rooms are untested.
• Decisions are frame-local. Long-horizon dialogue state belongs in your orchestrator.

License

Apache 2.0.

Citation

@misc{kupe2026thinkspark_realtime,
  title  = {Kupe-ThinkSpark-Realtime-270M: A Realtime Conversation Referee for Voice Agents},
  author = {Kupe},
  year   = {2026},
  url    = {https://huggingface.co/anuj-inavlabs/Kupe-ThinkSpark-Realtime-270M}
}

---

_{Built by Kupe}