Sophon-Realtime-32B-Tag

Code and one-command deployment: github.com/adventists-ai/sophon-realtime/tree/main/tag · Built on DuplexJev · Voice: tag-tts

The Tag build of Sophon-Realtime: it decides how to say it, and runs on one GPU. The 32B thinker listens to you, decides when it is its turn, and writes the reply together with inline sound tags: a laugh after a joke, a sigh for bad news, stress on the word that matters. A streaming voice (tag-tts) performs those tags while the reply is still being written. Tags are never read aloud and are removed from the text stream.

The whole system is one vLLM-Omni model: thinker → tag-tts talker → flow + HiFT vocoder, on a single 80 GB GPU.

Sophon-Realtime v1 family

Three builds share one brain (the same 32B model listens, decides turns and writes the reply). They differ only in how the reply becomes sound. Full comparison: github.com/adventists-ai/sophon-realtime.

	Tag (recommended)	Flash	E2E (preview)
How it speaks	an expressive speech model performs the sound tags	a lighter, faster speech model speaks the tagged reply	a talker inside the 32B model, no separate speech model
NVV-SuperBench (en / zh)	0.80 / 0.86	0.63 / 0.77	0.60 / 0.77
First audio, one user	~0.6 s	0.31 s	~1 s
Users per 80 GB GPU (recommended)	16	40	4–5
GPUs	1	1	2
Weights	Sophon-Realtime-32B-Tag	Sophon-Realtime-32B-Flash	new build in testing, coming soon

Pick Tag if you want the voice to act; pick Flash if latency and users per GPU matter most. Same APIs.

Highlights

• VoiceBench 82.8 and Big Bench Audio 81.8%: the strongest brain among open full-duplex models. #1 among open full-duplex models on both; level with GPT-4o-mini-Audio on VoiceBench.
• Paralinguistic understanding 48.6, #1 overall: it hears your tone. Ahead of Gemini 3.8 Flash (46.9) and Qwen3-Omni (44.6) on LISTEN, StressTest and MSPB. It hears emotion, emphasis, sarcasm, laughter and sighs straight from your voice.
• The most expressive build: NVV-SuperBench 0.86 (Chinese) and 0.80 (English). The model decides when to laugh, sigh, breathe, clear its throat or stress a word; the voice performs it instead of reading the tag aloud.
• One 80 GB GPU, one command. Served by vLLM-Omni; up to 16 concurrent users per GPU, about 0.6 s to first audio.
• Full-Duplex-Bench v3 tool calling: Pass@1 48%, ahead of Grok (43%), with a 100% response rate. Live mode is compatible with the OpenAI GPT-Live interface.
• Real full duplex. It listens the whole time you talk and drafts its reply as you go; it picks up when you stop, stops when you cut in, and keeps going when you just say "mm-hm".
• Apache-2.0, free for commercial use. Fine-tune persona, tone and domain knowledge with ms-swift.

Benchmark methods and full comparison tables: v1 family home page. To just try it: talk to it in the ModelScope Studio.

Deploy (one GPU, one command)

git clone https://github.com/adventists-ai/sophon-realtime && cd sophon-realtime/tag
bash install.sh                 # Python 3.12 venv, vLLM 0.30, vLLM-Omni 0.30, the plugin, and these weights (~42 GB)
GPU=0 PORT=8000 bash start.sh   # first start 15–20 min (kernels + TensorRT engines), later 5–8 min
python client_example.py --wav my_question.wav --lang en --voice ryan_en

• APIs: ws://HOST:8000/v1/realtime (full duplex: turn-taking, barge-in, backchannel) and POST /v1/chat/completions (audio in → text + audio stream).
• Voices: chelsie (default; chelsie_zh / chelsie_en picked by language), vivian_zh, ryan_en, brand_zh, brand_en.
• Hardware: one 80 GB GPU (H100 / H800 / A100 80G), about 65 GB per instance.

Performance (one H100 80 GB, measured)

Concurrent users (arriving within 10 s)	First audio, median / max	Stutter (median)
1	0.67 s	~0
8	0.61 / 0.76 s	0
16	0.75 / 1.07 s	0.06 s
24	1.36 / 2.11 s	0.39 s (queueing starts)

• Full duplex: about 0.7 s from the end of the user's speech to the first voice.
• Recommended: up to 16 concurrent users per GPU. More GPUs: one instance per GPU behind a load balancer; capacity scales roughly linearly. CPU is not a bottleneck (~1.5 cores per instance at 16 users).

Sound tags

[laughter], [breath], [sigh], [cough], <laughter>…</laughter> (laugh while speaking), <strong>…</strong> (stress a word). Tags are sent to the voice as whole chunks while text streams in, the same way the voice was trained.

Event heard when the tag is written (96 test sentences × 2 runs through the full served pipeline, AST detector; 8–12 sentences per class, so ±1 sentence ≈ ±10 points):

	sigh	breath	cough	laugh
this release	25%	50%	33%	42%
reference tag-tts renderer, same voice model	42%	50%	33%	50%

Live mode (OpenAI GPT-Live compatible)

Run one more command next to the engine and GPT-Live clients can connect by changing only the URL and key. Sophon keeps the voice; when a request needs a lookup or an action it says "Let me check that", hands the task to your backend and speaks the result in its own words.

cd sophon-realtime/live && BUILD=tag PORT=8000 LIVE_PORT=8780 bash start_live.sh   # GPT-Live clients: ws://HOST:8780/v1/live/sessions

Details and measurements: live/ in the GitHub repository.

Fine-tuning

Fine-tune the thinker's LLM backbone with LoRA in ms-swift to change persona, tone, domain knowledge and tool use. Hearing, the voice and turn-taking stay as released, and sound tags such as laughter and coughs are kept. One 80 GB GPU is enough; see finetune/ in the GitHub repository.

What's inside

folder	what	licence
thinker/	DuplexJev-32B-Para v1.5 with tag writing merged in, FP8 (with the vision tower)	Apache-2.0
talker/	tag-tts: language model + distilled flow (4 steps, no CFG) + HiFT; based on Fun-CosyVoice3-0.5B	Apache-2.0
voices/	six reference voices (clip + precomputed prompt)	Apache-2.0

Limitations

• [sigh] and <laughter>…</laughter> are still weak; the thinker writes them more often than the voice delivers.
• Mandarin is slightly less clear than English with the distilled flow; a laugh is sometimes transcribed as "哈哈" by ASR.
• Very long replies (over about a minute of speech in one turn) are beyond what the voice was trained on.