Surgical Emotion Module v3 — emotion2vec_plus_base @ 12.5 Hz (large-scale distillation)
Third generation of the surgical emotion module (v1: amir0907/student · v2: amir0907/student-v2). Same architecture and interface; trained on a ~23x larger, multi-corpus, multi-lingual pool.
Training data (v2 -> v3)
| v2 | v3 | |
|---|---|---|
| utterances | 2,160 (RAVDESS only) | 50,202 |
| corpora | RAVDESS speech+song | RAVDESS (2160) + CREMA-D (7442, en) + TESS (5600, en) + ESD (35000, en+zh) |
| steps | 2000 | 4000 (batch 8, ~32k samples seen) |
| teacher cache | 2 shards, 2x T4 | 4 shards, 4 workers on 2x T4 |
| held-out gate | RAVDESS actors 19-24 | same, never in any training pool |
Adapter implant (identical architecture, 3.18 M params):
LN -> Linear(768->512) -> 3x depthwise-separable conv (k=9, dil 1/2/4) -> grouped soft-attn pooling G=4 (50->12.5 Hz) -> LN -> Linear(512->768) -> heads.
Self-distillation from the frozen teacher — no labels used.
v1 / v2 / v3 on identical held-out inputs (RAVDESS actors 19-24 + zh clip, vs frozen teacher)
| metric | v1 | v2 | v3 |
|---|---|---|---|
| top-1 agreement | 100 % | 100 % | 100 % |
| mean JS divergence ↓ | 0.00035 | 0.00022 | 0.00027 |
| pooled-embedding cosine | 0.99935 | 0.99955 | 0.99948 |
| frame-level cosine (12.5->50 Hz) | 0.99524 | 0.99572 | 0.99560 |
| intensity dynamics (mean std) | 0.0301 | 0.0351 | 0.0214 |
| intensity corr | 0.310 | 0.326 | 0.397 |
Authoritative v3 numbers: reports/v1_v2_v3_compare.json (regenerate with
code/compare_v123.py). Rate contract re-checked at 1/2.5/5/10 s (exact 12.5 Hz).
Demo (demo/)
Per-sample folders with the actual test audio, the v3 analysis plot (frame emotion probabilities @12.5 Hz, intensity vs target, teacher-vs-v3 bars) and numeric results. Clips come from held-out RAVDESS actors (19-24) and the funasr Chinese test clip. RAVDESS audio is CC BY-NC-SA 4.0 (attribution: Livingstone & Russo 2018) — non-commercial use only; the zh clip is MIT.
Usage
from code.surgical_emotion import SurgicalEmotionEncoder
enc = SurgicalEmotionEncoder.from_pretrained(
"/path/to/emotion2vec_plus_base", # frozen teacher (funasr dir, Apache-2.0)
"surgical_emotion_module.pt", # this repo's checkpoint
device="cuda:0")
out = enc(wav_16k, wav_lengths)
# frame_embs (B,T,768) @12.5Hz | frame_probs (B,T,9) | frame_intensity (B,T,1)
# utt_emb (B,768) | utt_probs (B,9) | frame_lengths (B,)
Notes
- Inference cost unchanged vs v1/v2 (~2.7 ms adapter overhead; ~142x realtime total on T4).
- One-frame boundary tolerance comes from the teacher's conv frontend.
- Distilled on RAVDESS (CC BY-NC-SA) + CREMA-D/TESS/ESD: keep non-commercial or re-distill on permissive speech (label-free recipe, any corpus works).
- Part of an S2ST project:
frame_embsare designed for time-varying conditioning of dots.tts (2 emotion frames per 6.25 Hz LLM token).