Sky-Kim/com.sky.sentis.efficientat-mn04

sentis-efficientat-mn04

EfficientAT mn04_as (MobileNetV3, width 0.4) AudioSet audio tagger converted to Unity Inference Engine (Sentis) FP32. Recognizes 527 AudioSet sound classes (animals, alarms, vehicles, household and environmental sounds, music, human non-speech sounds, ...) from a raw mono waveform, and returns a 384-d embedding usable for your own classes.

Files

efficientat_mn04_fp32.sentis   # waveform [1, 96000] (32000 Hz mono, 3 s) -> scores [1, 527] (sigmoid), embedding [1, 384]
audioset_labels.txt   # 527 AudioSet display names, index order of the scores
EfficientAtMn04.cs   # self-contained Unity Sentis inference

Inference

The log-mel frontend is baked into the model (STFT as a fixed-basis Conv1d), so the whole pipeline waveform -> scores runs in one Sentis graph on CPU or GPU, with no feature extraction in C#. EfficientAtMn04 resamples any input rate to 32000 Hz, classifies the last 3 s, and also offers a sliding-window streaming buffer for microphone input.

A complete self-contained implementation lives in EfficientAtMn04.cs. Minimal usage:

var tagger = new EfficientAtMn04(BackendType.GPUCompute);
tagger.Load(modelRoot);              // folder holding efficientat_mn04_fp32.sentis + audioset_labels.txt
tagger.Warmup();

// one-shot: an AudioClip (any sample rate / channel count)
var data = new float[clip.samples * clip.channels];
clip.GetData(data, 0);
var scores = tagger.Classify(EfficientAtMn04.ToMono(data, clip.channels), clip.frequency);
foreach (var p in tagger.Top(scores, k: 5, minScore: 0.1f))
    Debug.Log($"{p.Label}: {p.Score:F2}");

// streaming: feed microphone chunks, classify the latest window every 0.25-0.5 s
tagger.Push(micChunk, micSampleRate);
var latest = await tagger.ClassifyLatestAsync();

Scores are independent sigmoid probabilities (multi-label), so several classes can be high at once; threshold per class (0.1-0.3 is a common starting point). tagger.Embedding holds the 384-d embedding of the last call for training a small classifier on your own sound classes.

Conversion notes

• Source: mn04_as_mAP_432.pt from the EfficientAT GitHub release v0.0.1.
• Architecture: MobileNetV3 (width multiplier 0.4) distilled from a PaSST transformer ensemble on AudioSet; 0.98M parameters.
• Frontend: 128-bin log-mel, 32 kHz, n_fft 1024 / window 800 / hop 320, pre-emphasis 0.97, (log(mel + 1e-5) + 4.5) / 5 (EfficientAT AugmentMelSTFT, eval mode). Reimplemented as a Conv1d with a windowed DFT basis + mel matrix taken from the original library, so no STFT / complex op is needed; it matches the original feature extractor.
• Exported with torch.onnx.export (opset 17) as one waveform -> (scores, embedding) graph at a fixed 3 s window and imported into Sentis 2.6.1.
• The exported graph matches the original PyTorch pipeline in ONNX Runtime, and Unity Sentis output matches ONNX Runtime on the same input (CPU and GPUCompute).
• The model was trained on 10 s clips: windows shorter than ~2 s lose noticeable accuracy, so keep the 3 s window (Push + ClassifyLatest) rather than classifying very short snippets.

License & attribution

MIT. Converted from EfficientAT by Florian Schmid et al. (https://github.com/fschmid56/EfficientAT). Paper: Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillation (ICASSP 2023). AudioSet class names: AudioSet ontology (CC BY 4.0).