Sky-Kim/com.sky.sentis.ced-tiny

sentis-ced-tiny

CED-Tiny (Consistent Ensemble Distillation ViT) AudioSet audio tagger converted to Unity Inference Engine (Sentis) FP32. Recognizes 527 AudioSet sound classes (animals, alarms, vehicles, household and environmental sounds, music, human non-speech sounds, ...) from a raw mono waveform, and returns a 192-d embedding usable for your own classes.

Files

ced_tiny_fp32.sentis   # waveform [1, 48000] (16000 Hz mono, 3 s) -> scores [1, 527] (sigmoid), embedding [1, 192]
audioset_labels.txt   # 527 AudioSet display names, index order of the scores
CedTiny.cs   # self-contained Unity Sentis inference

Inference

The log-mel frontend is baked into the model (STFT as a fixed-basis Conv1d), so the whole pipeline waveform -> scores runs in one Sentis graph on CPU or GPU, with no feature extraction in C#. CedTiny resamples any input rate to 16000 Hz, classifies the last 3 s, and also offers a sliding-window streaming buffer for microphone input.

A complete self-contained implementation lives in CedTiny.cs. Minimal usage:

var tagger = new CedTiny(BackendType.GPUCompute);
tagger.Load(modelRoot);              // folder holding ced_tiny_fp32.sentis + audioset_labels.txt
tagger.Warmup();

// one-shot: an AudioClip (any sample rate / channel count)
var data = new float[clip.samples * clip.channels];
clip.GetData(data, 0);
var scores = tagger.Classify(CedTiny.ToMono(data, clip.channels), clip.frequency);
foreach (var p in tagger.Top(scores, k: 5, minScore: 0.1f))
    Debug.Log($"{p.Label}: {p.Score:F2}");

// streaming: feed microphone chunks, classify the latest window every 0.25-0.5 s
tagger.Push(micChunk, micSampleRate);
var latest = await tagger.ClassifyLatestAsync();

Scores are independent sigmoid probabilities (multi-label), so several classes can be high at once; threshold per class (0.1-0.3 is a common starting point). tagger.Embedding holds the 192-d embedding of the last call for training a small classifier on your own sound classes.

Conversion notes

• Source: mispeech/ced-tiny (model.safetensors, re-exported from PyTorch; the repo's own model.onnx is int8-quantized and does not import into Sentis).
• Architecture: ViT (12 layers, dim 192, 3 heads, 16x16 patches) trained with consistent ensemble distillation on AudioSet; 5.5M parameters.
• Frontend: 64-bin log-mel, 16 kHz, n_fft 512 / window 512 / hop 160, power to dB with top_db = 120 (CED CedFeatureExtractor). Reimplemented as a Conv1d with a windowed DFT basis + mel matrix taken from the original library, so no STFT / complex op is needed; it matches the original feature extractor.
• Exported with torch.onnx.export (opset 17) as one waveform -> (scores, embedding) graph at a fixed 3 s window and imported into Sentis 2.6.1.
• The exported graph matches the original PyTorch pipeline in ONNX Runtime, and Unity Sentis output matches ONNX Runtime on the same input (CPU and GPUCompute).
• CED splits inputs longer than 10 s into chunks; the 3 s window used here stays within one chunk, so the graph is the plain ViT forward (time positional embeddings for the first 18 patches).
• The model was trained on 10 s clips: windows shorter than ~2 s lose noticeable accuracy, so keep the 3 s window (Push + ClassifyLatest) rather than classifying very short snippets.

License & attribution

Apache License 2.0. Converted from CED by Heinrich Dinkel et al. (https://github.com/RicherMans/CED, https://huggingface.co/mispeech/ced-tiny). Paper: CED: Consistent ensemble distillation for audio tagging (ICASSP 2024). AudioSet class names: AudioSet ontology (CC BY 4.0).