cris-cmd/michi-audio-affect

Michi Audio Affect

An English speech-emotion classifier built for the fast listener in Michi. It classifies 16 kHz mono speech into six coarse vocal-tone labels:

Label	Meaning
ang	anger
dis	disgust
fea	fear
hap	happiness
neu	neutral
sad	sadness

The repository contains the quantized ONNX runtime artifact. It does not contain training audio.

Intended use

This model is intended as a low-latency, secondary signal for conversational presentation. Michi combines its output with text emotion and deterministic rules; the model never changes plans, prices, reservations, or other application state.

Do not use it to diagnose mental health, infer stable personal traits, make high-impact decisions, or treat a predicted emotion as a fact about a person.

Training

• Base encoder: facebook/wav2vec2-base, frozen at revision 0b5b8e8
• Dataset: myleslinder/crema-d, revision 8a11ae8
• Input: 16 kHz speech audio
• Feature: mean-pooled hidden state from encoder layer 7
• Head: multinomial logistic regression
• Split: actor-disjoint; eight actors held out for validation and the two local fixture actors excluded from training
• Export: Wav2Vec2ForSequenceClassification → ONNX → dynamic int8 quantization

The dependency- and revision-pinned build recipe is in scripts/train-audio-affect-head.py in the Michi repository. It is intended to be repeatable, but does not promise bit-for-bit identical artifacts across operating systems and hardware.

Validation

The logistic-regression head reached 78.2% accuracy across six classes on an actor-disjoint validation split. The same validation split selected the best of 13 encoder layers, so this is a model-selection result rather than an estimate from an untouched test set. The score was calculated before ONNX export; the quantized artifact has runtime behavior checks, but has not been evaluated over the complete validation split.

Class	Recall
anger	95.5%
disgust	77.7%
fear	63.4%
happiness	73.2%
neutral	93.8%
sadness	67.9%

The pre-export confusion matrix, validation actor IDs, and source revisions are in TRAINING_REPORT.json.

Usage with Transformers.js

import { pipeline } from "@huggingface/transformers";

const classify = await pipeline(
  "audio-classification",
  "cris-cmd/michi-audio-affect",
  { dtype: "q8" },
);

// Float32Array containing mono, 16 kHz PCM samples.
const scores = await classify(audio, { top_k: null });

Michi pins a downloaded copy under models/affect/michi-audio-affect so it can run without a model-host request during a conversation.

Limitations

• Trained on acted English speech using a small set of fixed sentences.
• The six labels are intentionally coarse and do not cover natural emotional complexity.
• Accuracy will vary with language, accent, microphone quality, background noise, and spontaneous speech.
• Treat scores as uncertain presentation hints and combine them with text and explicit user intent.

Licenses and attribution

The exported model is released under Apache-2.0. The base encoder is Apache-2.0. CREMA-D is distributed under the Open Database License, with individual contents covered by the Database Contents License. Users are responsible for complying with the source dataset terms.

CREMA-D citation:

@article{cao2014crema,
  title={CREMA-D: Crowd-sourced Emotional Multimodal Actors Dataset},
  author={Cao, Houwei and Cooper, David G. and Keutmann, Michael K. and Gur, Ruben C. and Nenkova, Ani and Verma, Ragini},
  journal={IEEE Transactions on Affective Computing},
  volume={5},
  number={4},
  pages={377--390},
  year={2014},
  doi={10.1109/TAFFC.2014.2336244}
}