emotions_speech — speaker-independent speech emotion recognition

A wav2vec2-large-XLSR-53 fine-tune that classifies emotional tone from raw audio alone — from how something is said, not what is said.

Classes (7): neutral · happy · sad · angry · fearful · disgust · surprised

The evaluation protocol is the point

This model is evaluated with speaker-independent GroupKFold, and the split is asserted at runtime rather than merely intended:

assert len(train_actors & test_actors) == 0, "Speaker leakage!"

A random split places the same actors in both train and test, so the model learns to recognise voices rather than emotions. That is how speech-emotion models reach the 90%-plus accuracy figures commonly reported for this task — and why those figures collapse on the first unseen speaker.

Per-fold metrics live in the training notebook. No headline figure is quoted here: the only figures worth quoting are the speaker-disjoint ones, and a single averaged number hides how much fold-to-fold variance a 7-class problem over a small pool of actors actually carries.

Architecture

Beyond a plain classification head:

• Learned weighted sum over all 25 transformer hidden states — nn.Parameter(torch.ones(25)) — instead of taking only the final layer. Emotion is carried largely in the mid-layers, and letting the model learn the mixture beats guessing which layer to read.
• Projected residual path: LayerNorm → 768→384 → GELU → 384→128, summed with a linear proj_skip shortcut before the classifier.
• Input: raw 16 kHz waveform, 4-second clips, up to 6 chunks aggregated per turn.

Because the pooling layer is custom, this checkpoint does not load through a stock AutoModelForAudioClassification call.

Usage

Training

• Data: RAVDESS, split speaker-independently with GroupKFold
• Loss: focal loss, to handle class imbalance
• Regularisation: mixup, label smoothing
• Schedule: cosine annealing with warm restarts
• Augmentation: ESC-50 environmental noise
• Second view: a complementary EfficientNet-B2 over multi-channel spectrograms (mel + MFCC + delta-MFCC) was trained for ensembling

Intended use

Conversational systems that need vocal tone as a signal independent of transcript content.

Deployed in Solace, where it runs alongside a facial-expression classifier and the two channels are deliberately kept separate. When a user's wording reads neutral while their voice reads fearful, that disagreement is surfaced to the user rather than averaged into a single score — the gap is the most informative thing the system can notice, and averaging destroys it.

Limitations

• RAVDESS is North American English acted speech. Acted emotion is more exaggerated than spontaneous emotion, and the cultural distribution does not match Tunisian or other non-Western speakers. Expect degradation outside that distribution.
• Seven categorical labels are a coarse model of affect. Real emotion is continuous, mixed, and context-dependent.
• Outputs are classifier predictions, not measurements of a person's internal state. Any system built on this should present them that way — "your tone read as frustrated", never "you were frustrated".
• Not for clinical, diagnostic, hiring, or surveillance use.

Related work

Model	Purpose
Ghazouaniwala/silma-tts-derja	Tunisian Derja speech synthesis — the other half of Solace's speech stack
Ghazouaniwala/trocr-tunisian-arabic	Handwritten Tunisian Arabic recognition

Citation

@misc{ghazouani2025emotionsspeech,
  author = {Ghazouani, Wala Eddine},
  title  = {emotions_speech: speaker-independent speech emotion recognition},
  year   = {2025},
  url    = {https://huggingface.co/Ghazouaniwala/emotions_speech}
}

---

Built by Wala Eddine Ghazouani · GitHub · LinkedIn