Roxi-Duplex: a voice agent you can interrupt, in Indian English

Most voice agents take turns: you speak, you wait, they speak. Roxi-Duplex does not wait. It is a full-duplex speech-to-speech model that listens while it talks, so callers can interrupt it mid-sentence, murmur "haan, okay" while it speaks, and get a response in a natural Indian-English voice that opens with "Welcome to Voz Vox. My name is Roxi. How may I help you today?"

To our knowledge this is one of the first openly released Indian-English adaptations of a full-duplex speech-to-speech model. It is a compact LoRA adapter (370 MB) for Kyutai's Moshi 7B, so you get frontier duplex behavior plus an Indian support-agent persona without downloading a new foundation model.

Why you might want this

• True barge-in. Duplex is architectural, not a VAD hack: the model tracks both audio streams on one timeline, so interruptions and backchannels work the way they do between humans.
• Indian-English out of the box. Accent and delivery learned from Indian speakers, for the hundreds of millions of users that US-accented agents serve poorly.
• Support-agent behavior built in. Greetings, bookings, order status, complaints, payments. Prompt it with a caller and it answers like a call-center agent, not a chatbot.
• Real time on one GPU. Moshi streams at a real-time factor of about 0.34 on an A100 40 GB, with about 200 ms theoretical latency. The adapter adds nothing to inference cost.
• A reproducible recipe. Open Indian full-duplex dialogue data does not exist, so we synthesized it. The full pipeline is documented below and cheap to rerun (training takes 17 minutes on one A100).

Quick facts

Field	Value
Base model	kyutai/moshiko-pytorch-bf16 (7B, CC-BY 4.0)
Audio codec	Mimi (streaming, 12.5 Hz frames, 24 kHz)
Method	LoRA rank 64, scaling 2.0, 1500 steps, via Kyutai moshi-finetune
Adapter size	370 MB (safetensors)
Training data	150 synthetic two-channel support conversations, about 94 minutes
Assistant channel	Roxi TTS (Indian-English, 1.7B MOSS-TTS-Local fine-tune) speaking scripted support dialogues
User channel	Real Indian-English speakers from ai4bharat/Svarah (117 speakers, CC-BY 4.0)
Training cost	About 17 minutes on one rented A100 40 GB

How the data was made

There is no open Indian-English full-duplex dialogue corpus, so we built one:

1. Scripted VozVox support dialogues (greetings, bookings, complaints, payments) with Indian names, cities, and numbers written as words.
2. Assistant turns rendered with an Indian-English TTS voice (Roxi), silence-trimmed and time-stretched 1.25x with WSOLA for a natural pace. WSOLA matters: phase-vocoder stretching made the voice sound robotic.
3. User turns taken from real Svarah recordings across India.
4. Both sides placed on a shared stereo timeline with turn gaps, backchannels, and overlaps, plus word-level alignments for Moshi's inner-monologue text stream.

Usage

Requires a Linux GPU with the moshi package (Triton is needed for the real-time compiled path, so native Windows is not supported for real-time use).

pip install moshi

import torch
from huggingface_hub import hf_hub_download
from moshi.models.loaders import CheckpointInfo, get_lora_moshi
from moshi.models import LMGen

adapter = hf_hub_download("IOTEverythin/roxi-duplex", "lora.safetensors")
info = CheckpointInfo.from_hf_repo("kyutai/moshiko-pytorch-bf16")
mimi = info.get_mimi(device="cuda")
lm = info.get_moshi(device="cuda", dtype=torch.bfloat16)
lm = get_lora_moshi(lm, adapter, 64, 2.0,
                    dtype=torch.bfloat16, device="cuda", fuse_lora=True)
lm_gen = LMGen(lm, temp=0.7, temp_text=0.7)
# Stream user audio through mimi.encode and lm_gen.step exactly as with base Moshi.

Two tips from our experiments:

• Prompt the model with real user audio. Feeding only silence makes any Moshi-family model produce unfocused speech.
• If you fine-tune further, stay light. Around rank 64 and 1500 steps was the sweet spot; heavier adapters (rank 96, 3000 steps) kept the accent but degraded intelligibility.

Limitations

• Proof-of-concept scale: 150 synthetic conversations from four scripted scenario templates. Coverage outside customer-support topics is limited.
• The dialogue structure is synthetic; real-call turn-taking dynamics may differ.
• English only (Indian-English accent); no Hindi code-switching yet.
• Inherits all Moshi limitations and requires a GPU for real-time use.

License and attribution

Released under CC-BY 4.0, matching the base model. This work builds on:

• Moshi and Mimi by Kyutai (kyutai/moshiko-pytorch-bf16, CC-BY 4.0). Defossez et al., "Moshi: a speech-text foundation model for real-time dialogue".
• Svarah by AI4Bharat (ai4bharat/Svarah, CC-BY 4.0), used for the user audio channel.
• The assistant voice derives from our Roxi TTS, fine-tuned on the IIT-Madras Indic TTS English set. Required notice: COPYRIGHT 2016 TTS Consortium, TDIL, Meity, represented by Hema A. Murthy and S. Umesh, Department of Computer Science and Engineering and Electrical Engineering, IIT Madras. ALL RIGHTS RESERVED.
• Trained with Kyutai's moshi-finetune.

See also our Indian-English TTS models: IOTEverythin/roxi-tts-pro (1.7B premium) and IOTEverythin/roxi-tts-v3.1 (0.1B real-time).

Responsible use

This model speaks with a synthetic voice derived from consented and licensed datasets. Do not use it to impersonate real people or for fraud, social engineering, or deception. Disclose AI-generated audio where required by law or policy. Provided as is, without warranty.