Surgical Emotion Module — emotion2vec_plus_base @ 12.5 Hz
A surgically modified emotion2vec_plus_base that outputs continuous, frame-level emotion vectors at 12.5 Hz (80 ms/frame) instead of a single pooled utterance vector — designed as the emotion-conditioning front-end of a Speech-to-Speech Translation (S2ST) system built on top of dots.tts (its LLM token rate is 6.25 Hz, so exactly 2 emotion frames align to every LLM token).
What the surgery changes
| stock emotion2vec_plus_base | this module | |
|---|---|---|
| output | 1 pooled 768-d utterance vector | per-frame 768-d continuous vectors @ 12.5 Hz |
| emotion probs | utterance-level, 9 classes | per-frame 9 classes + utterance-level |
| intensity | none | continuous [0,1] per frame |
| dynamics | discarded by pooling | preserved (within-word intensity changes) |
Architecture of the implant (3.18 M trainable params, the 93.8 M backbone stays frozen):
frozen emotion2vec_plus_base backbone (50 Hz frame states, 768-d)
-> LayerNorm -> Linear(768->512)
-> 3x depthwise-separable conv fusion (k=9, dilation 1/2/4)
-> grouped soft-attention pooling over G=4 frames (50 Hz -> 12.5 Hz)
-> LayerNorm -> Linear(512->768) (continuous emotion vectors)
-> heads: emotion logits (9) + intensity (sigmoid)
Intensity is distilled from a confidence-gated expression-energy target (sqrt(max-prob) x utterance-normalized RMS envelope), which gives real temporal dynamics instead of the saturated confidence proxy.
Outputs
out = enc(wav_16k, wav_lengths)
out["frame_embs"] # (B, T, 768) continuous emotion vectors @ 12.5 Hz
out["frame_probs"] # (B, T, 9) per-frame emotion probabilities
out["frame_intensity"] # (B, T, 1) per-frame intensity in [0,1]
out["utt_emb"] # (B, 768) pooled utterance embedding
out["utt_probs"] # (B, 9)
out["frame_lengths"] # (B,)
# T = ceil(teacher_frames / 4); frame i covers audio [i*0.08, (i+1)*0.08) s
Usage
Requires funasr and the frozen teacher backbone (Apache-2.0, from ModelScope/HF:
iic/emotion2vec_plus_base), then:
from code.surgical_emotion import SurgicalEmotionEncoder
enc = SurgicalEmotionEncoder.from_pretrained(
"/path/to/emotion2vec_plus_base", # frozen teacher (funasr model dir)
"surgical_emotion_module.pt", # this repo's checkpoint
device="cuda:0")
out = enc(wav_batch_16k, wav_lengths)
See code/ for the full pipeline: module definition, self-distillation training,
evaluation, teacher-vs-student benchmark, and demo-gallery generation.
Training
Self-distillation from the frozen teacher — no emotion labels needed:
- losses: frame cosine + LN-MSE (student upsampled to 50 Hz vs teacher states), utterance cosine, utterance KL, per-frame KL, intensity MSE
- data: RAVDESS speech, actors 1-18 (900 utterances); actors 19-24 held out
- 900 steps, AdamW lr 3e-4, fp16, single Tesla T4 (~5 min)
Results (held-out actors 19-24)
| metric | value |
|---|---|
| top-1 agreement with teacher | 100 % (56/56 files, all 8 emotions) |
| mean JS divergence vs teacher | 0.0010 |
| pooled-embedding cosine | 0.9994 |
| frame-level cosine (12.5->50 Hz reconstruction) | 0.9956 |
| rate contract | exact 12.5 Hz (+-1 frame boundary) |
Benchmark (T4, fp32): total student path 34.1 ms/utt (142x realtime, RTF 0.0096);
surgical adapter overhead only 2.7 ms (3.4 % params, ~2.3 % latency). Adapter
scales to 2395 utt/s at batch 8; fp16 autocast gives another ~22 %. Full numbers
in reports/benchmark_report.json.
demo/ contains per-sample analysis plots (frame probabilities @12.5 Hz,
intensity vs target, teacher-vs-student bars) and numeric results for 7 samples
including the cross-lingual funasr Chinese test clip.
Notes
- One-frame boundary tolerance at utterance end comes from the teacher's conv frontend (inherent to emotion2vec, not the surgery).
- The adapter was distilled on RAVDESS (CC BY-NC-SA 4.0): keep usage non-commercial or re-distill on permissively licensed speech (the recipe is label-free, so any emotional speech corpus works).
- Part of an S2ST project: next step injects
frame_embsinto dots.tts as a time-varying condition alongside its speaker x-vector path.