amir0907/student-v2

Surgical Emotion Module v2 — emotion2vec_plus_base @ 12.5 Hz

Second, larger-data distillation of the surgical emotion module (successor of amir0907/student, now called v1). Same architecture; trained with 2.4x more data and 2.2x more steps.

Outputs continuous, frame-level emotion vectors at 12.5 Hz (80 ms/frame) plus per-frame 9-class probabilities and a continuous intensity in [0,1] — designed as the emotion-conditioning front-end of an S2ST system on dots.tts (6.25 Hz LLM tokens -> exactly 2 emotion frames per token).

Training data (v1 -> v2)

	v1	v2
utterances (self-distill, no labels)	900 (speech, capped)	2160 (speech + song)
teacher states	frozen emotion2vec_plus_base, 50 Hz, 768-d	same
optimization	900 steps, AdamW 3e-4	2000 steps, AdamW 3e-4, warmup 100
hardware	1x T4	teacher cache precomputed in parallel on 2x T4
train actors (RAVDESS)	1-18	1-18 (speech 1080 + song 1080)
held-out test	actors 19-24 speech	same, never seen

Adapter implant (identical architecture): 3.18 M params, LN -> Linear(768->512) -> 3x depthwise-separable conv (k=9, dil 1/2/4) -> grouped soft-attn pooling G=4 (50->12.5 Hz) -> LN -> Linear(512->768) -> heads.

v1 vs v2 on identical held-out inputs (32 files, actors 19-24, vs frozen teacher)

metric	v1	v2
top-1 agreement	100 %	100 %
mean JS divergence	0.00035	0.00022
pooled-embedding cosine	0.99935	0.99955
frame-level cosine (12.5->50 Hz)	0.99524	0.99572
intensity dynamics (mean std)	0.0301	0.0351
intensity corr vs conf-gated energy	0.310	0.326

v2 matches or beats v1 on every quality metric while retaining the exact 12.5 Hz rate contract (checked at 1/2.5/5/10 s). Inference cost is unchanged (~2.7 ms adapter overhead on top of the frozen backbone; T4 fp32).

Full per-file comparison: reports/v1_v2_compare.json · run log: reports/train_v2.log

Usage

from code.surgical_emotion import SurgicalEmotionEncoder

enc = SurgicalEmotionEncoder.from_pretrained(
    "/path/to/emotion2vec_plus_base",   # frozen teacher (funasr dir, Apache-2.0)
    "surgical_emotion_module.pt",       # this repo's checkpoint
    device="cuda:0")
out = enc(wav_16k, wav_lengths)
# out["frame_embs"] (B,T,768) @12.5Hz | frame_probs (B,T,9)
# frame_intensity (B,T,1) | utt_emb (B,768) | utt_probs (B,9) | frame_lengths (B,)

code/ contains the full pipeline: module, parallel teacher-cache precompute (precompute_shard.py), distillation (train_v2.py), evaluation, benchmark, demo-gallery builder, and the v1-vs-v2 comparison script.

Notes

• One-frame boundary tolerance at utterance end comes from the teacher's conv frontend (inherent to emotion2vec, not the surgery).
• Distilled on RAVDESS (CC BY-NC-SA 4.0): keep usage non-commercial or re-distill on permissively licensed speech (the recipe is label-free).
• Part of an S2ST project: next step injects frame_embs into dots.tts as a time-varying condition alongside its speaker x-vector path.