Hinglish Turn Detection β v2 (s2-004)
Tiny audio-native turn detector for Hinglish (HindiβEnglish code-mixed) telephony speech: given up to 8 s of audio, is the user done speaking (complete β agent can respond) or just pausing (incomplete β keep listening)? Two-stage fine-tune of the Smart Turn v3.2 approach for the language pair it skipped.
Results
| Model | test-A (v3.2-test hin+eng, 1,200) | test-B (TTS Hinglish, 33) | test-C (MUCS real speech, 846) |
|---|---|---|---|
| Smart Turn v3.2 zero-shot | 0.927 / 0.930 | 0.788 / 0.759 | 0.539 / 0.573 |
| v2 s2-004 (this model) | 0.882 / 0.886 | 0.970 / 0.968 | 0.600 / 0.446 |
(acc / F1). The intended story: the baseline wins on its own training distribution; the Hinglish fine-tune wins on Hinglish domains. int8 variant: β2.5 / β3.0 (1 clip) / +0.4 acc, 9.05 MB, p50 ~13 ms CPU. Policy (P6 sweep): Ο=0.5 default; knee Οβ0.45; test-B flat Ο 0.25β0.85. Trailing context: accuracy saturates at ~4 s (test-A) / 2 s (test-B).
Input contract (identical to Smart Turn)
- 16 kHz mono float32 in [-1, 1]
- β€ 8 s: shorter β zero-pad FRONT; longer β keep the LAST 8 s
- Feature convention: whisper log-mel (80, 800) via
chunk_length=8WITHOUT per-bin normalization (do_normalize=False). The v2 models were trained and exported self-consistently on unnormalized mels (transformers 5 silently dropped the FE default). Do NOT enable normalization at inference. - Output: 1 logit β apply sigmoid; β₯ 0.5 β complete (at the shipped Ο=0.5)
import numpy as np, onnxruntime as ort, soundfile as sf
from huggingface_hub import hf_hub_download
from transformers import WhisperFeatureExtractor
fe = WhisperFeatureExtractor.from_pretrained("openai/whisper-tiny", chunk_length=8)
sess = ort.InferenceSession(hf_hub_download("Shrey160/hinglish-turn-v2", "onnx/s2-004.fp32.onnx"))
audio, sr = sf.read("clip.wav", dtype="float32") # resample to 16 kHz mono first
feats = fe(audio, sampling_rate=16000, return_tensors="np", padding="max_length",
max_length=8 * 16000, truncation=True, do_normalize=False)["input_features"]
logit = sess.run(None, {sess.get_inputs()[0].name: feats.astype(np.float32)})[0].reshape(-1)[0]
p_complete = 1 / (1 + np.exp(-logit))
Training (two-stage transfer, whisper-tiny encoder)
- Stage 1 β frozen encoder, head trained on original Smart Turn hin+eng (7,200 clips), lr 5e-5, 4 epochs.
- Stage 2 β 232 TTS Hinglish clips (Γ2 upsample) + 50:50 hin/eng replay, lr 1e-4, 3 epochs, last 2 encoder blocks unfrozen (approved amendment). Ablations (ASP pooling, attention-end, label smoothing, unfreeze k, replay ratio) and KD (whisper-small teacher) all lost to this default or shipped as documented negatives.
Limitations
- test-C is MUCS tutorial monologue β all neural models collapse there (prosody domain gap); treat as stress test, relative rankings only.
- test-B is 33 clips sharing the TTS pipeline with s2 training data.
- mels are unnormalized (see above); normalized retraining = future work.
- 83% of the v3.2 training pool is TTS-generated; human Hinglish data is the top scaling lever.