Sky-Kim/com.sky.sentis.yamnet

sentis-yamnet

YAMNet (Google, MobileNetV1) AudioSet audio tagger converted to Unity Inference Engine (Sentis) FP32. Recognizes 521 AudioSet sound classes (animals, alarms, vehicles, household and environmental sounds, music, human non-speech sounds, ...) from a raw mono waveform, and returns a 1024-d embedding usable for your own classes.

Files

yamnet_fp32.sentis   # waveform [1, 15600] (16000 Hz mono, 0.975 s) -> scores [1, 521] (sigmoid), embedding [1, 1024]
audioset_labels.txt   # 521 AudioSet display names, index order of the scores
Yamnet.cs   # self-contained Unity Sentis inference

Inference

The log-mel frontend is baked into the model (STFT as a fixed-basis Conv1d), so the whole pipeline waveform -> scores runs in one Sentis graph on CPU or GPU, with no feature extraction in C#. Yamnet resamples any input rate to 16000 Hz, classifies the last 0.975 s, and also offers a sliding-window streaming buffer for microphone input.

A complete self-contained implementation lives in Yamnet.cs. Minimal usage:

var tagger = new Yamnet(BackendType.GPUCompute);
tagger.Load(modelRoot);              // folder holding yamnet_fp32.sentis + audioset_labels.txt
tagger.Warmup();

// one-shot: an AudioClip (any sample rate / channel count)
var data = new float[clip.samples * clip.channels];
clip.GetData(data, 0);
var scores = tagger.Classify(Yamnet.ToMono(data, clip.channels), clip.frequency);
foreach (var p in tagger.Top(scores, k: 5, minScore: 0.1f))
    Debug.Log($"{p.Label}: {p.Score:F2}");

// streaming: feed microphone chunks, classify the latest window every 0.25-0.5 s
tagger.Push(micChunk, micSampleRate);
var latest = await tagger.ClassifyLatestAsync();

Scores are independent sigmoid probabilities (multi-label), so several classes can be high at once; threshold per class (0.1-0.3 is a common starting point). tagger.Embedding holds the 1024-d embedding of the last call for training a small classifier on your own sound classes.

Conversion notes

• Source: the official Keras weights yamnet.h5 (https://storage.googleapis.com/audioset/yamnet.h5) with the model definition from tensorflow/models research/audioset/yamnet, rebuilt in PyTorch (the TF Hub / Kaggle TFLite build is int8-quantized and computes its STFT with RFFT2D).
• Architecture: MobileNetV1 (depthwise-separable convolutions, ReLU) trained on AudioSet; 521 classes; 3.7M parameters.
• Frontend: 64-bin log-mel, 16 kHz, 25 ms window / 10 ms hop (FFT 512), periodic Hann, 125-7500 Hz, magnitude spectrogram, log(mel + 0.001) (YAMNet features.py), one 96-frame patch. Reimplemented as a Conv1d with a windowed DFT basis + mel matrix taken from the original library, so no STFT / complex op is needed; it matches the original feature extractor.
• Exported with torch.onnx.export (opset 17) as one waveform -> (scores, embedding) graph at a fixed 0.975 s window and imported into Sentis 2.6.1.
• The exported graph matches the original Keras (TensorFlow) model in ONNX Runtime, and Unity Sentis output matches ONNX Runtime on the same input (CPU and GPUCompute).
• One inference = one YAMNet patch (96 frames = 0.96 s of log-mel from 0.975 s of audio), exactly the unit the original yamnet_frames_model scores; slide it with Push + ClassifyLatest (YAMNet itself hops 0.48 s).
• TensorFlow same padding (asymmetric for stride 2) and the TF STFT framing (no centering, window left-aligned in the 512-point FFT) are reproduced exactly; scores match the original Keras model.

License & attribution

Apache License 2.0. Converted from YAMNet by Google / TensorFlow Model Garden (https://github.com/tensorflow/models/tree/master/research/audioset/yamnet). Paper: CNN Architectures for Large-Scale Audio Classification (ICASSP 2017; YAMNet applies the MobileNetV1 architecture, arXiv:1704.04861). AudioSet class names: AudioSet ontology (CC BY 4.0).