amir0907/student-v5

Surgical Emotion Module v5 — 6.25 Hz + weak supervision from REAL intensity labels

Fifth generation of the surgical emotion module (v1: amir0907/student · v2: amir0907/student-v2 · v3: amir0907/student-v3 · v4: amir0907/student-v4).

Same interface as v4 (frozen emotion2vec_plus_base backbone + 3.18 M-param adapter, 6.25 Hz = exactly 1 emotion frame per dots.tts LLM token, salience-gated absolute-energy intensity target, corr+std losses). v5 adds the one thing v4 proved was missing: the label-free intensity proxy had a hard ceiling (the target itself separated RAVDESS normal/strong by only Cohen-d 0.23, and the v4 student sat at 0.26). v5 breaks through it with weak supervision from the intensity labels that already existed in the training corpora but were never used:

• RAVDESS (speech+song): filename field 4 — 01=normal, 02=strong → 0.0 / 1.0
• CREMA-D: filename suffix — LO=0.0, MD=0.5, HI=1.0 (XX skipped)
• 3,941 labeled utterances of the 50,618-utt pool (2,576 RAVDESS + 1,365 CREMA-D)

v5 supervision

Pairwise margin ranking on the utterance-mean intensity (mean of the 6.25 Hz frame intensities): for every labeled pair with label_i > label_j, loss = relu(margin − (pred_i − pred_j)), margin 0.10. The term is shift-invariant, so it cannot fight the frame-level distillation target — it only enforces ORDERING (strong > normal, HI > MD > LO). Every 2nd batch is drawn from the labeled subset so pairs are frequent. All other losses/recipe identical to v4; same cache; 4000 AdamW steps, batch 8, fp16, single T4 (~14 min).

Results — identical held-out inputs (RAVDESS actors 19-24 + zh clip, vs frozen teacher)

metric	v1	v2	v4	v5
top-1 agreement	100 %	100 %	100 %	100 %
mean JS divergence ↓	0.00035	0.00022	0.00044	0.00067
frame-level cosine (up to 50 Hz)	0.99524	0.99572	0.99198	0.99294
intensity corr (new target)	0.237	0.239	0.594	0.629
intensity curve std (dynamics)	0.030	0.035	0.183	0.171
normal(01) vs strong(02) mean-intensity diff	+0.021	+0.027	+0.047	+0.063
intensity Cohen-d (336 emotional files)	0.57	0.66	0.26	0.34
target's own Cohen-d (ceiling)	—	—	0.23	0.34 — ceiling exceeded
CREMA-D LO/MD/HI Spearman (in-train)	0.30	0.33	0.29	0.32
per-emotion strong>normal	—	—	—	5/7

Rate contract exact at 6.25 Hz for 1/2.5/5/10 s inputs. Authoritative numbers: reports/v1_v2_v4_v5_compare.json, reports/eval_report_v5.json.

Honest notes:

• v5 is the first student whose intensity ordering exceeds what its own label-free target encodes — the supervision term did exactly what it was added for. Biggest per-emotion gains: happy +0.18, sad +0.20; calm and surprised remain ≈ 0 (their strong/normal contrast is inherently subtle).
• The CREMA-D Spearman is roughly flat across versions (~0.3): the LO/MD/HI labels are subjective and partially confounded with recording level. The decisive evidence for v5 is the held-out 01/02 separation + corr.
• JS/frame-cos stay at the same (tiny) level as v4 — the ordering gain cost nothing in emotion fidelity.

Demo (demo/)

9 samples: one strong-intensity utterance per emotion (03-08) + neutral + the funasr zh clip — each with the actual test audio, the v5 analysis plot and numeric results. RAVDESS audio is CC BY-NC-SA 4.0 (Livingstone & Russo 2018) — non-commercial use only; the zh clip is MIT.

Usage

from code.surgical_emotion import SurgicalEmotionEncoder

enc = SurgicalEmotionEncoder.from_pretrained(
    "/path/to/emotion2vec_plus_base",   # frozen teacher (funasr dir, Apache-2.0)
    "surgical_emotion_module.pt",       # this repo's checkpoint (G=8 restored automatically)
    device="cuda:0")
out = enc(wav_16k, wav_lengths)
# frame_embs (B,T,768) @6.25Hz | frame_probs (B,T,9) | frame_intensity (B,T,1)
# utt_emb (B,768) | utt_probs (B,9) | frame_lengths (B,)
# T = ceil(duration_s * 6.25); frame i covers audio [i*0.16, (i+1)*0.16) s

Notes

• Adapter identical to v1-v4 (3.18 M params, ~2.7 ms overhead); no re-precompute was needed — labels are parsed from filenames at cache-load time.
• Distilled on RAVDESS (CC BY-NC-SA) + CREMA-D/TESS/ESD: keep non-commercial or re-distill on permissive speech.
• Part of an S2ST project: frame_embs are designed for per-token time-varying conditioning of dots.tts (1 emotion frame per 6.25 Hz LLM token).