Sky-Kim/com.sky.sentis.bcresnet-8

sentis-bcresnet-8

BC-ResNet-8 (Broadcasted Residual Network, base width 64) environmental sound classifier for Unity Inference Engine (Sentis) FP32. Classifies a raw mono waveform into the 50 ESC-50 environmental sound classes (animals, natural soundscapes, human non-speech sounds, domestic and urban noises). The network has only 331K parameters; it was trained from scratch on ESC-50 because no AudioSet-pretrained weights exist for this architecture.

Files

bcresnet8_esc50_fp32.sentis   # waveform [1, 80000] (16000 Hz mono, 5 s) -> logits [1, 50]
esc50_labels.txt   # 50 ESC-50 category names, index order of the scores
BcResNet8.cs   # self-contained Unity Sentis inference

Inference

The log-mel frontend is baked into the model (STFT as a fixed-basis Conv1d), so the whole pipeline waveform -> logits runs in one Sentis graph on CPU or GPU, with no feature extraction in C#. BcResNet8 resamples any input rate to 16000 Hz, classifies the last 5 s (the ESC-50 clip length) and returns softmax probabilities; it also offers a sliding-window streaming buffer for microphone input.

A complete self-contained implementation lives in BcResNet8.cs. Minimal usage:

var classifier = new BcResNet8(BackendType.GPUCompute);
classifier.Load(modelRoot);          // folder holding bcresnet8_esc50_fp32.sentis + esc50_labels.txt
classifier.Warmup();

var data = new float[clip.samples * clip.channels];
clip.GetData(data, 0);
var probs = classifier.Classify(BcResNet8.ToMono(data, clip.channels), clip.frequency);
foreach (var p in classifier.Top(probs, k: 3))
    Debug.Log($"{p.Label}: {p.Score:F2}");

// streaming: feed microphone chunks, classify the latest window periodically
classifier.Push(micChunk, micSampleRate);
var latest = await classifier.ClassifyLatestAsync();

The classes are mutually exclusive (softmax), so there is always a winner: add your own "background / none" handling, e.g. require a minimum top probability. To recognize your own sounds, retrain the network on your data with the same frontend.

Training & conversion notes

• Architecture: BC-ResNet-8 (broadcasted residual blocks with SubSpectral Normalization, base width 64), 50-class head; 331K parameters.
• Frontend: 40-bin log-mel, 16 kHz, n_fft 512 / window 480 / hop 160, periodic Hann, log(mel + 1e-6) (BC-ResNet LogMel, torchaudio). Implemented as a Conv1d with a windowed DFT basis + mel matrix taken from the original library, matching the original feature extractor; the network was trained on exactly these features.
• Training: ESC-50 folds 1-4 (fold 5 held out), 120 epochs, AdamW + one-cycle schedule, mixup (alpha 0.3) and time/frequency masking, random 4 s crops; evaluated on full 5 s clips.
• Exported with torch.onnx.export (opset 17) as one waveform -> logits graph at a fixed 5 s window and imported into Sentis 2.6.1. Unity Sentis output matches ONNX Runtime on the same input (CPU and GPUCompute).
• AdaptiveAvgPool2d((1, None)) (frequency pooling inside each BC-ResBlock) is exported as a mean over the frequency axis (identical values).

License & attribution

Model weights: CC BY-NC 3.0 (trained on ESC-50, CC BY-NC 3.0 - non-commercial use only); wrapper code (Runtime/): MIT. Architecture: BC-ResNet by Qualcomm AI Research (https://github.com/Qualcomm-AI-research/bcresnet) (BSD-3-Clause-Clear). Paper: Broadcasted Residual Learning for Efficient Keyword Spotting (Interspeech 2021). Dataset: K. J. Piczak, ESC: Dataset for Environmental Sound Classification (ACM MM 2015).