Index-Echo-S2ST-2B

🌐 Online demo · GitHub · 📚 Technical report · 🤗 Hugging Face collection · ModelScope collection

Index-Echo-S2ST-2B translates a Chinese, Japanese, or English speech clip into another language and generates speech conditioned on the source speaker's voice. It combines a frozen Index-Echo-S2TT-2B-family backbone, a learned Hidden2CV mapper, and CosyVoice3 speech generation in one self-contained package and Python process.

The release supports twelve directions: Chinese → English/Spanish/Japanese/Korean, Japanese → English/Spanish/Korean/Chinese, and English → Chinese/Spanish/Japanese/Korean. The caller specifies the target language; the package infers the source language from its source transcript. 2B denotes the speech translator's decoder size, rather than the total size of all packaged components.

This v4.1 package supersedes the initial six-direction release: it adds Japanese source input, Korean target output, and Japanese → Chinese, and its mapper is tuned with one round of rejection-sampling fine-tuning (RFT) on top of distillation.

Architecture

The report's lower path extends S2TT to S2ST: the frozen speech translator supplies its final hidden states to Hidden2CV, which connects to CosyVoice3 speech generation. Source audio supplies the reference prompt and CampPlus speaker embedding for voice conditioning. The S2TT backbone remains frozen throughout S2ST training.

Model and training

The technical report initializes S2ST from the trained Index-Echo S2TT model, whose Qwen3-Omni AuT audio encoder and connector feed an Index-Translate decoder. The speech-generation path uses a roughly 26M-parameter Hidden2CV mapper to connect the translator's final hidden states to the CosyVoice3 semantic layer.

Training first distills the mapper with both the S2TT backbone and speech generator frozen, then applies a round of rejection-sampling fine-tuning (RFT): best-of-G samples are drawn from the distilled mapper's own synthesis pool, scored, and replayed with ground-truth anchoring. The S2TT backbone remains frozen throughout. At inference, source audio supplies the reference prompt and CampPlus speaker embedding for voice conditioning. Voice similarity varies with the source, target language, and synthesis result.

The translator in this package is the EXP-59 checkpoint, whose continued pre-training adds Japanese-source speech understanding on top of the released S2TT recipe; a 40/40 probe set passes for both genuine English and Japanese source audio.

Inference

Download the complete model package before importing its local modeling_dubbing.py:

pip install -U huggingface_hub
hf download IndexTeam/Index-Echo-S2ST-2B --local-dir ./Index-Echo-S2ST-2B
cd Index-Echo-S2ST-2B

Pinned environment

The release documents the following tested dependency pins. Use CUDA-compatible PyTorch wheels; the +cu129 builds require a wheel source that provides that CUDA build. System ffmpeg must be on PATH.

pip install torch==2.11.0+cu129 torchaudio==2.11.0+cu129 transformers==5.6.0 \
    tokenizers==0.22.2 librosa==1.0.0 onnxruntime==1.30.0 wetext==0.0.4 \
    kaldifst==1.8.0 conformer==0.3.2 hydra-core==1.3.7 HyperPyYAML==1.2.3 \
    lightning==2.6.6 x-transformers==2.11.24 einx==0.4.3 frozendict==2.4.7 \
    pyworld==0.3.5 soundfile==0.13.1 modelscope==1.37.1 safetensors==0.7.0 \
    numpy==2.4.6 openai-whisper==20250625

openai-whisper is used only for the package's ASR-based evaluation/self-check. The Japanese frontend (pyopenjtalk==0.4.1, including its dictionary, and pykakasi==2.3.0) is vendored under code/ja_ext/; the pipeline adds it to the import path automatically. All components run in one Python environment and process.

The package automatically applies two CosyVoice3 compatibility patches for Transformers 5.6 in code/tts/pipeline.py: keeping the CosyVoice3 LLM in fp32, and providing the full visible attention mask during autoregressive decoding. Keep the bundled code and dependency pins together.

Python API

Run this from the downloaded package directory, with input clips available at the indicated paths:

from modeling_dubbing import DubbingBridgeModel

model = DubbingBridgeModel.from_pretrained('.', device='cuda')

# Chinese source: four supported targets.
wav, sr = model.dub('input_zh.wav', lang='en', out_wav='dub_en.wav')
model.dub('input_zh.wav', lang='ko', out_wav='dub_ko.wav')

# Japanese source: four supported targets, including Chinese.
model.dub('input_ja.wav', lang='zh', out_wav='dub_zh.wav')
model.dub('input_ja.wav', lang='en', out_wav='dub_en2.wav')

# English source: four supported targets.
model.dub('input_en.wav', lang='ja', out_wav='dub_ja.wav')
model.dub('input_en.wav', lang='es', out_wav='dub_es.wav')

wav, sr, info = model.dub('input_ja.wav', lang='en', return_info=True)
print(sr)               # 24000 Hz
print(info['src_lang']) # Detected source language: 'zh', 'ja', or 'en'
print(info['tgt_raw'])  # Model translation, before the speech frontend's normalization.
print(info['tgt_cv'])   # Speech-frontend text; Japanese uses a katakana representation.

lang always means the target language: en, es, ja, ko, or zh. The source language is detected from the transcript of the source audio: a transcript containing kana is treated as Japanese, one containing CJK ideographs as Chinese, and otherwise English; it is not a general language identifier. Source and target must differ; zh→zh and en→en raise an error. The waveform is a float32 tensor of shape (1, T) at 24 kHz. out_wav saves it to a file. With return_info=True, the third return value includes the transcript, translation, source language, and synthesis diagnostics.

English-source input must be genuine English speech: heavily accented synthetic English (for example, TTS English read with a Chinese accent) can make the translator's transcript collapse into kana. That is a translator hearing boundary, not a mapper fault.

At the lower pipeline layer, extract(audio_path, lang) returns the translated text and aligned hidden states separately from synthesis. See the S2ST inference guide for the dub.py wrapper, and the video pipeline for segmentation and video timeline alignment.

Default inference settings

Stage or setting	Released default
Transcription and translation	Greedy, temperature=0.0 / do_sample=False
Transcription/translation budget	max_new_tokens=1024
Speech mode	h2cv (the only supported mode in this package)
Speech-token budget	max_tokens=1500; actual cap min(20 * m, 1500), where m is aligned target-text token count
Minimum speech length	2 * m tokens before stop tokens are allowed
Speech sampling	sampling=25; release notes describe ras_sampling, top-k 25 / top-p 0.8
Speech-generation seed	42
Synthesis speed	1.0
Output sample rate	24000 Hz
CosyVoice3 execution	stream=False, load_trt=False, load_vllm=False, fp16=False; LLM kept in fp32
Chunked orchestration	Disabled (chunk=False)

Greedy transcription/translation does not make speech synthesis greedy. The public DubbingBridgeModel.dub wrapper does not expose a seed argument; seed changes are available through the lower-level pipeline's dub/synth APIs. Sampling may vary between calls even with the default seed, as documented in the release notes.

Configurable API parameters

The public wrapper and lower pipeline expose different controls.

Interface	Parameter	Default	Behavior
from_pretrained	model_dir	Required	Local directory containing the full package
from_pretrained	device	cuda	CUDA device for the loaded components
from_pretrained	mode	None → config default h2cv	Sets the default speech mode
Public dub	audio_path	Required	Chinese, Japanese, or English source clip
Public dub	lang	en	Target language: en, es, ja, ko, zh; must differ from detected source
Public dub	out_wav	None	Save path; None creates a temporary WAV
Public dub	mode	None → loaded default	Only h2cv is accepted
Public dub	chunk	False	True is unsupported in this release
Public dub	return_info	False	Adds transcript, translation, and synthesis diagnostics to the return value
Pipeline extract	lang	en	Target language for transcription/translation and alignment
Pipeline extract	max_new_tokens	1024	Transcription/translation output budget
Pipeline synth	out_wav	None	Optional output save path
Pipeline synth	max_tokens	1500	Upper bound on generated speech tokens
Pipeline synth	seed	42	Seed set before speech-token generation and acoustic synthesis
Pipeline synth	prompt_wav	None	Uses the source audio unless a reference clip is supplied
Pipeline synth	prompt_text	None	Uses the source transcript unless reference text is supplied

For explicit text/speech budgets or seed control, use the lower pipeline after loading model as above:

pipe = model._pipe
ext = pipe.extract('input_zh.wav', lang='en', max_new_tokens=1024)
wav, info = pipe.synth(ext, out_wav='dub_en.wav', max_tokens=1500, seed=42)

The lower pipeline's dub(..., **kw) forwards synthesis options such as seed and max_tokens; changing the translation budget requires calling extract separately. Sampling argument 25 and synthesis speed 1.0 are fixed in the released pipeline code, not keyword arguments of the public wrapper. Translation stops at tokenizer EOS or <|im_end|>; speech generation uses CosyVoice3's defined stop tokens and the length limits above. The script does not explicitly override other text-generation knobs such as top_p, top_k, or repetition penalties.

Evaluation

Content fidelity is measured per direction by reading the synthesized speech back with whisper-large-v3 and comparing it against the translator's own target text: WER for English and Spanish targets, CER for Japanese, Korean, and Chinese targets (median, n=200 per axis; held-out cohorts excluded from mapper training). This caliber measures acoustic fidelity and controllability of the speech-generation path — the reference translation is produced by this chain's own speech translator, so it is not an independent translation-quality benchmark. These are different metrics and test sets; do not compare values directly across target languages.

Direction	Median		Direction	Median
zh→en	0.036 WER		ja→zh	0.050 CER
zh→es	0.062 WER		en→zh	0.000 CER
zh→ja	0.059 CER		en→es	0.111 WER
zh→ko	0.020 CER		en→ja	0.000 CER
ja→en	0.037 WER		en→ko	0.000 CER
ja→es	0.051 WER		ja→ko	0.020 CER

Against the distilled (pre-RFT) mapper, the RFT round improves the 22-axis paired macro trim-10 mean from 0.0683 to 0.0612 (−10.4%), with three directions significantly better (zh→ko, ja→zh, en→ko) and no direction significantly regressed under per-utterance paired testing. Compared with the initial six-direction release on the same cohort, all six shared directions are significantly better (note the reference translations favor this chain, so treat that comparison as directional evidence). Full numbers are recorded in config.json under eval_s2_metrics.

Package details

File or directory	Purpose
config.json, modeling_dubbing.py	Component configuration and DubbingBridgeModel interface
stlm_llm/	Qwen3.5-family translator decoder and tokenizer (EXP-59)
stlm_ckpt/	Trained audio encoder/connector checkpoint
stlm_omni/	Qwen3-Omni configuration and AuT source components
cosyvoice3/	CosyVoice3 LLM, flow, HiFT, and ONNX assets
bridge/	mapper.safetensors, mapper_config.json (5 language slots), and mapper_best_5slot.pt checkpoint archive
code/bridge/ko_norm.py	Korean text normalization (character-level; ASCII digits to Korean number words)
wetext_en_tn/, wetext_repo/	Offline text-normalization assets
code/tts/, code/stlm/, code/bridge/	Dubbing, speech translation, and mapper implementation
code/ja_ext/, code/cosyvoice_repo/	Vendored Japanese frontend and CosyVoice inference code
samples/, verify_export.py	Input clips, reference dubs, and export/self-check assets

The loader prefers safetensors for the translator, mapper, and CosyVoice3 components, with original-format fallback. The export records version v4.1.0-ml-s2full12-v4rft-stlm59 (2026-10-12), translator EXP-59 iter14535, and mapper h2cv_ml_s2_full12_v4/mapper_best_5slot (distilled, then one RFT round). Keep bundled assets together for offline loading. Use materialized copies (cp -rL or an appropriate rsync configuration) when copying training-side packages whose weight files are hard links.

Text normalization and alignment

Direction	Translator text used for alignment	Speech-frontend text	Hidden-state alignment
zh→en	Normalized English	Same normalized text	Character-overlap mean pooling
zh→es	Normalized Spanish	Same normalized text	Character-overlap mean pooling
zh→ja	Original mixed-script Japanese	Katakana frontend representation	Word mapping to original text, then pooling
zh→ko	Original translation	Normalized Korean (character-level)	Coordinate remap to original text, then pooling
ja→*	Same per-target rules as zh source
en→zh	Original translation	Wetext-normalized Chinese	Character-overlap mean pooling
en→es	Original translation	Normalized Spanish	Character-overlap mean pooling
en→ja	Original mixed-script Japanese	Katakana frontend representation	Word mapping to original text, then pooling
en→ko	Original translation	Normalized Korean (character-level)	Coordinate remap to original text, then pooling

The source transcript is the first nonempty line and the translation is the last nonempty line of translator output. The mapper's language-slot list is ['en', 'es', 'ja', 'ko', 'zh'] with indices en=0, es=1, ja=2, ko=3, zh=4; the loader asserts this mapping. Do not reorder it when modifying or converting the package. (The 5-slot mapper is a numerically exact trim of the 13-slot training mapper — forward outputs match with max abs diff 0.0.)

Limitations

• Use one CUDA GPU with at least 12 GB VRAM. CPU inference is unverified; memory use depends on the runtime and input.
• The release is tuned for utterance-level dubbing. Keep individual utterances at or below approximately 30 seconds; use the full video pipeline for longer recordings. chunk=True raises NotImplementedError in this release.
• Japanese → Japanese is outside the direction contract but remains runnable: its median CER (0.063–0.068) is close to the Chinese-source baseline, yet about 9–10% of samples collapse into ultra-short mapper outputs that trigger a CosyVoice3 convolution error (kernel size > input size). For production Japanese output, use zh→ja or en→ja.
• Same-language pairs zh→zh and en→en are unsupported (extract raises an error); for dubbing in the source language, use the source audio directly.
• Repetition loops are an inherent failure mode of the hidden2cv mapper (as in all previous packages) and occur occasionally on long, difficult sentences.
• English-source input must be genuine English speech; heavily accented synthetic English can break the translator's transcript (see the Inference section).
• Translation errors propagate into speech. Inspect tgt_raw and tgt_cv when fidelity is important.
• Source-voice conditioning does not guarantee identical timbre, pronunciation, or delivery. The source-language heuristic can misclassify atypical or mixed-language transcripts.
• The release notes report occasional failures on roughly 1–2% of tested short English-source inputs, including a CosyVoice convolution-kernel error or a one-token collapse producing approximately 0.04 seconds of audio. Check the generated audio and retry through the lower-level pipeline with a different seed, or discard the failed result.

Related models

Family	Task	Released sizes
Index-Translate	Text translation and translation instructions across 150 languages	2B, 9B, 35B-A3B (preview)
Index-Echo S2TT	Speech-to-text translation and subtitles	2B, 9B
Index-Echo S2ST	Speech-to-speech translation with source-voice conditioning	2B, 9B
Index-Homura	Translation with a target syllable count	2B, 9B
Index-NativeLong	Native long-document translation; released as Index-Nailong	2B, 9B

The text foundation's 150-language coverage does not describe the released speech interfaces. Use the task-specific directions documented above.

Citation

@techreport{indextranslate2026,
  author={Tianjiao Li and Mengran Yu and Chenyu Shi and Lusheng Zhang and
          Qisi Chen and Yanshan Zhou and Ji Qi and Jingying Liu and
          Yuang Feng and Ziang Cui and Tianxing Yan},
  title={Index-Translate: A Multilingual Translation Model Family --- Text, Speech, Controlled Dubbing, and Long-Document Translation},
  institution={Index LLM Team},
  year={2026},
  month={September}
}

License and feedback

Apache-2.0. Questions and feedback are welcome through GitHub Issues.