siba · full-duplex speech for Hindi, Telugu and English

siba is a speech-to-speech dialogue model that listens and speaks at the same time, built by adapting Kyutai's Moshi to Hindi and Telugu while keeping English. This repository holds the current Stage 2b checkpoint plus the Stage 1 base checkpoints it was built from.

Model	Moshiko (~7.7B parameters, Mimi codec) with a 48k text vocabulary (+16k Hindi/Telugu pieces)
Languages	Hindi, Telugu, English
Training	Stage 1: ~33k h YouTube monologue speech → Stage 2b: 517 h two-channel dialogues, 1,500 steps
Agent voices	one fixed voice per language (Hindi: Ritvik)
Status	research checkpoint — turn-taking works, answer relevance and response speed do not yet
Weights	stage2b/checkpoint_001500/consolidated/consolidated.safetensors (15.7 GB, bf16)

Highlights

• Listens before it speaks. In Full-Duplex-Bench v1 it stays quiet through 78–81% of the user's mid-sentence pauses, where the original Moshi talks over every one, and it talks over the user in 25% of cases versus 89%.
• Right language, right role. After dialogue training it replies in the language of the conversation and as an assistant; the Stage 1 model before it monologues, often in the wrong language.
• Still slow and often off-topic. It takes 1.4–2 s to respond and often does not answer what was asked, especially in Hindi and Telugu (see Limitations).

Results

Full-Duplex-Bench v1 (English)

	siba Stage 2b	Moshiko
Duplex score (0–100)	38.4	54.5
Talks over the user in mid-sentence pauses ↓	0.19 – 0.22	1.00
Talk-over share of all cases ↓	0.25	0.89
Takes the turn when the user finishes ↑	0.92	1.00
Latency to take the turn ↓	1.36 s	0.00 s*
Yields when interrupted ↑ / latency ↓	0.94 / 2.00 s	0.92 / 0.68 s
Backchannel frequency ↑	0.06	0.16

25% subset: the first 36 samples of each of the 5 tasks (180), both models run with the same streaming inference. The duplex score is a timing-only composite of the four components (each takes the takeover rate and gives full latency credit at ≤0.3 s, none at ≥2 s); the official GPT relevance judge was not run. *Moshi's 0 s latency means it was already talking — its higher score partly rewards that.

Held-out loss

Step	Hindi	Telugu	English
0 (Stage 1, before dialogue training)	7.94	7.69	7.71
500	0.835	0.925	0.572
1500 (released)	0.807	0.904	0.562

48 held-out one-minute units per language. No overfitting: the final step is the best.

Quick start

pip install moshi huggingface_hub

import torch
from huggingface_hub import snapshot_download
from moshi.models import loaders, LMGen

d = snapshot_download("BNarayanaReddy/siba", allow_patterns=["stage2b/checkpoint_001500/*", "stage2b/tokenizer.model"])
info = loaders.CheckpointInfo.from_hf_repo(
    "kyutai/moshiko-pytorch-bf16",                     # Mimi codec
    moshi_weights=f"{d}/stage2b/checkpoint_001500/consolidated/consolidated.safetensors",
    tokenizer=f"{d}/stage2b/tokenizer.model",
    config_path=f"{d}/stage2b/checkpoint_001500/consolidated/config.json",
)
model = info.get_moshi(device="cuda", dtype=torch.bfloat16)
mimi = info.get_mimi(device="cuda"); mimi.set_num_codebooks(8)
lm_gen = LMGen(model, temp=0.8, temp_text=0.7, top_k=250, top_k_text=25)
# Streaming as in Moshi: Mimi-encode the user's audio one 80 ms frame (1920 samples, 24 kHz) at a time;
# lm_gen.step(codes) returns the model's text token + audio codes for that frame; mimi.decode() gives its speech.

Model details

Training data — 45,204 synthetic two-channel dialogues, 517 h

Language	Train hours	Script sources	Agent voice	User voices (vary per dialogue)
Hindi	211	samvaad-hi-v1, indic-align (Wiki_Conv, OpenAssistant_T), snorbyte	Ritvik	120 IndicVoices speakers (60 male / 60 female)
Telugu	178	indic-align (Wiki_Conv, OpenAssistant_T), Updesh, snorbyte	m3_te_s38	120 IndicVoices speakers (60 / 60)
English	128	SODA, OpenAssistant	m3_en_s0	100 Svarah Indian-English speakers (50 / 50)

Scripts were cleaned (prompts and reasoning stripped, replies capped at ~50 words), voiced with the Maya 3 TTS, force-aligned (MMS) and assembled into two-channel conversations: median turn gap ~0.3 s, ~25% overlapping turn changes, barge-ins where the agent yields within ~0.27 s, and backchannels from a per-voice bank. Audio is Mimi-encoded at 8 codebooks per channel; 473 dialogues (5.6 h) are held out.

Training setup — 1,500 steps on 7 × H100, 31 min

Setting	Value
Init	Stage 1, step 15k (stage1/checkpoint_015000)
Fine-tuning	full, Mimi frozen
Batch	49 × 60 s (0.82 h of audio per step), ~2.4 passes over the data
Optimizer	AdamW, LR 2e-5, one-cycle with 5% warm-up, weight decay 0.1
Loss	agent text + agent audio (first codebook ×100, text padding ×0.5); the user channel is input only
Language weights	Hindi 0.35 · Telugu 0.35 · English 0.30

Exact config: stage2b/args.yaml · eval log: stage2b/stage_eval.jsonl · benchmark reports: stage2b/bench/

Stage 1 (the base)

Monologue speech from YouTube — Hindi 12.8k h, Telugu 12.1k h, English 8.4k h — with the user channel silent, so it is not conversational on its own. Checkpoints are in stage1/ (5k, 10k, 15k; a continuation to 30k is in progress). Stage 2b starts from step 15k.

Limitations

• Relevance. Replies are often fluent but not about what the user asked, most of all in Hindi and Telugu. Understanding the user's speech is weak: Stage 1 never had a user channel and Stage 2b adds only ~500 h of listening. Hindi/Telugu knowledge answers can invent facts.
• Latency. It responds 1.4–2 s after the user finishes, against ~0.3 s gaps in its training data.
• Synthetic data only. All voices are TTS and the timing is rule-based; no real Hindi/Telugu conversations. Long-answer sources (OpenAssistant, Updesh) give few exchanges per minute.
• Evaluation. Benchmark numbers are English only and from a 25% subset (components can move by ±0.1).
• Audio. Moshi's 8-codebook Mimi on top of TTS audio limits clarity.

Repository contents

All folders

Folder	What it is
stage2b/	this model: weights, config, tokenizer, training config, eval log, benchmark reports, card images
stage1/	Stage 1 checkpoints with eval logs
moshiko_ext16k/	the untrained starting point: Moshiko with the text vocabulary extended 32k → 48k
tokenizers/	tokenizer candidates (table below)
configs/, scripts/, results/	training configs, data and training scripts, evaluation reports

moshiko_ext16k. Ids 0–31999 are Moshi's original pieces (weights unchanged; English tokenises to identical ids); ids 32000–47999 are Telugu/Hindi pieces whose embedding and output rows start as the mean of the byte tokens they replace. The start-of-text row moved to 48000. On original-vocab input, audio logits and the first 32,000 text logits are bit-identical to Moshiko. Apply Unicode NFC to text before tokenising.

Tokenizer	Vocab	en tokens/word	hi tokens/word	te tokens/word
original Moshi	32,000	1.28	10.70	19.66
extend_16k (used)	48,000	1.278	1.23	1.53
extend_24k	56,000	1.278	1.19	1.43
replace_32k	32,000	1.34	1.20	1.46

License and attribution

Weights derive from Kyutai's Moshiko (CC-BY 4.0; copyright Kyutai, modified as described above). Training scripts come from SODA (CC-BY 4.0), sarvamai/samvaad-hi-v1 (Apache-2.0), ai4bharat indic-align, microsoft/Updesh and snorbyte samples (CC-BY 4.0); user voices were cloned from AI4Bharat's IndicVoices and Svarah. Check each source's terms before uses beyond research.