Austin207/voice-fan-controller-model

Model card — dscnn_fan int8 (voice-fan-controller)

The deployed model: 18-class spoken-command recognition (16 fan commands + _silence_ + _unknown_) running on a plain ESP32-WROOM-32. 28,584 bytes, int8, no accelerator. Input is a (81, 32) log-mel plane = 2,592 floats; output is an 18-way softmax.

The flashed image does not run TensorFlow Lite Micro. It runs a generated static-C engine (dscnn_micro) that reproduces the same graph — verified against the TFLite interpreter at 0 prediction flips over 200 clips, pre-softmax logits within 1 LSB. A TFLM build still exists behind the same C API and is selectable at menuconfig, but it is not what ships.

The hard requirement is speaker independence — it has to work for someone who never recorded a clip. Everything below is organised around how well it does that, and around not quoting numbers that measure something easier.

The retired func_mlp model card is at archive/MODEL_CARD_func_mlp.md in this model repo (docs/archive/MODEL_CARD.md in the code repo). Its accuracy numbers do not apply to this model — they are in-speaker.

The number that matters

Leave-one-speaker-out (LOSO) — score each speaker with a model that never heard them. It is the only honest metric here, and it belongs to a recipe, not to a weights file: the shipping model trains on everybody, so it has no held-out voice of its own left to be measured against.

The deployed model's own recipe (c48_rio15 — all voices, two-stage transfer with BN frozen, domain_aug=0.5, encoder widened to 48 channels; measured 2026-07-28) reaches 91.69% headline, 5 seeds per fold:

anirudh	austin	francis	mishelle	headline
91.34%	93.86%	95.55%	86.00%	91.69%

That is +3.25pp over the last func_mlp-era shipping recipe (88.44%), against 1 SE of 1.81pp — clear of the pre-registered accept bar (≥2pp and ≥1 SE).

Against the immediately preceding recipe (c48_deployable, 91.37%) it is only +0.32pp — inside noise. The two differ by corpus, not by code: three more speakers' worth of recordings (+710 clips). It is reported as a corpus refresh, not as a lever that cleared a bar.

Eleven further complete speakers are scored outside the headline. Averaged over all fifteen full-coverage speakers — the widest "what a stranger gets" view — it reads 90.05% (SE 2.27pp), and its worst voice is 66.85% (aak). Read the worst, not the mean: this refresh raised the average while lowering the hardest voice by 1.98pp, which is precisely the failure mode the worst-voice rule exists to catch. The full spread, weakest first:

aak 66.9 · savio 74.0 · mishelle 86.0 · rio 86.5 · allen 90.5
anirudh 91.3 · adithyasharma 92.9 · alwin 94.1 · austin 93.9 · anon1 93.3
anon2 95.0 · danushkrishna 95.4 · francis 95.6 · adithya 97.0 · teresa 98.4

The width helped studio voices and hurt browser ones. Widening the encoder raised every studio-corpus fold (+1.53 to +5.64pp) but lowered all three small browser-recorded folds (adithyasharma −5.93, anon1 −2.25, anon2 −6.22). That is roughly 14pp of fold-sum, and it is the whole reason this model lands just under 90%. A headline win does not automatically transfer to strangers on a laptop mic.

A matched control run separates the two things that changed: the extra recorded data is worth +0.88pp on its own (a reject as a lever), and capacity alone is +2.04pp. The width is the real lever, and it clears the 2pp bar narrowly — not by a comfortable margin.

(The fifteen-speaker average is not comparable to older fourteen- or seven-speaker figures — the corpus grew, so the same statistic now covers a different population. Only the frozen headline four are comparable across runs.)

In-speaker validation for the deployed model is 97.47% and is not a generalization estimate. Every held-out clip there belongs to a speaker who is in training — by design, since nobody is held out of the shipping model. The gap between the two kinds of number is the entire point of this project: an earlier architecture scored ~95% in-speaker and 43.7% LOSO. If you quote one number, quote a LOSO one.

Limitations, in order of how likely they are to bite

1. Pronunciation, not channel, is what sinks the worst voice. aak sits at 66.85% — twenty-three points below the mean, and moving the wrong way across the last two recipes — and the errors are within-word confusions, not noise. A channel probe put that speaker at z=0.98 (unremarkable recording conditions), and cepstral mean normalisation made things worse by 2.8pp, so the gap is genuinely acoustic: those pronunciations land near the wrong class in feature space. More dense recordings of the weak voices is the only lever with evidence behind it; roughly eight code-side levers have been tried and spent.
2. Vocal register is the second axis. Nearly every training speaker sits at 107–149 Hz median F0; one sits at 255.7 Hz, roughly an octave up. That speaker is still the weakest headline fold (86.00%) and is the only one register augmentation helps. Expect degradation for voices outside the trained range, which includes most children. The headline averages three men and one woman, so it understates this failure mode.
3. English only, one accent pool, 18 speakers. The encoder is pretrained on 9,286 outside speakers, but fine-tuning corpus diversity is small — and four of the scored fifteen contributed only ~16–20 clips each, so their coverage is nominal rather than substantial. Those tiny folds are honest (genuinely held out) but noisy: one misheard clip moves such a fold by ~6pp.
4. timer_* is the weakest command group. Public pretraining corpora contain ten instances of the word "timer" in total, so transfer cannot teach it — it is learned only from the recorded set.
5. Siblings dominate the errors, and the gate cannot catch them. 66.7% of all command errors confuse two parameters of the same command (power_on↔power_off, speed_2↔speed_3, timer_2h↔timer_4h) — errors that produce a plausible-looking wrong action rather than an obvious one. This was priced on silicon (see Downstream gate): the wrong answers are often delivered with high confidence, so no threshold separates them from correct ones.
6. _unknown_ has zero human speakers — it is sourced from YouTube audio. Rejection of speech that is not a command is therefore weaker than the negative-class accuracy suggests.

Feature contract — not optional

sample_rate 20 kHz · window 25.6 ms (n_fft 512) · stride 20 ms
mel_bins 32 · log-mel via power_to_db(ref=max, top_db=80) · input (81, 32, 1)

ref=max is a correctness requirement, not a tuning choice. The alternative (log(mel + 1e-6)) has no amplitude normalisation, so raw int16 PCM on the device shifts every feature by ~6σ of pure recording level — frozen BatchNorm cannot absorb it and the model runs while predicting nonsense. window 25.6 ms gives n_fft = 512 so the device can use a radix-2 FFT.

The C frontend is measured amplitude-invariant (0.25× gain moves features by 0.0016 dB) and agrees with the training frontend to <0.002 dB.

Training

DS-CNN (depthwise-separable conv stack, 672,672 MACs, 12,082 params, 48 channels). The encoder is pretrained on Speech Commands v2 + MSWC — 112,049 clips from 9,286 speakers, 28.1% female — then fine-tuned on 5,542 recorded clips from 18 speakers.

The 48-channel width is a re-pretrain, not a knob: the encoder weights are shaped by channels, so widening it invalidates the previous encoder and forces a full pretraining run before any comparison is fair. The previous 32-channel model was 6,706 params / 441k MACs / 20,632 bytes.

Transfer is what makes this work: from-scratch on the same data scores 45.9%, against 67.4% pretrained. That +21.5pp is the single largest gain on record here, and nothing else has come within 20pp of it. (That pair was measured with an earlier encoder — dscnn_fan_scv2, 11,376 speakers — not the 9,286-speaker scv2_max above. It is quoted as evidence that pretraining is what carries this model, which is encoder-independent; it is not a measurement of the encoder shipping today.)

The fine-tune is two-stage: 15 epochs with the encoder frozen so the randomly initialised 18-class head cannot wash out the transferred filters, then unfreeze at lr 3e-4 with BatchNorm statistics frozen — those were estimated over 9,286 speakers and re-estimating them from a handful of fold voices reintroduces the per-channel speaker signature the pretraining was for.

Two train-time-only augmentations, neither of which costs anything on device:

• register augmentation (Praat "Change gender", moving F0 and formants independently) — covers the register axis the corpus barely spans;
• domain augmentation (reverb + bandlimit + µ-law codec round-trip) — added for the deployed model to close the browser-microphone gap. It has still never been isolated in a controlled run, so it remains a bet, and it is labelled as one. What is known is that the gap is not closed: the three browser-recorded folds were the only ones the wider encoder made worse.

Training is seeded but not bit-reproducible — oneDNN varies float reduction order across threads, moving single folds by up to 5.1pp between identical runs. Compare 5-seed means, not folds.

On device

Measured on a real ESP32-D0WD-V3 from the 2026-07-27 flashed image. That image carries the previous weights (c48_deployable); the model described by this card is built but not yet flashed. Every number in this section is a property of the graph, not of the weights — identical shapes, ops, parameter count and MACs — so it carries over unchanged. The accuracy numbers above do not: those belong to c48_rio15, and the board has not yet been asked to reproduce them.

		previous (c32, TFLM, single-core)
frontend	18.6 ms	45.3 ms
invoke	34.2 ms	49.75 ms
total	~53 ms (after a 1,600 ms capture window)	~95 ms
tensor arena	13,184 B (static engine)	17,324 B used / 20,480 reserved
flash	28,584 B	20,632 B

Do not credit that speedup to the model. This image is ~1.8× faster while computing +53% more MACs, because two other things changed at the same time: the static-C engine replaced TFLM, and both the STFT and the conv stack now use the second core. The wider model taken alone would have been slower. Two variables moved; the comparison is reported, not attributed.

The arena is unchanged at 13,184 B despite the wider layers, which is arithmetic rather than luck: the peak simultaneously-live tensor pair is the input features plus the stem output, and channels does not touch stem_channels=16.

Ops: CONV_2D, DEPTHWISE_CONV_2D, MAX_POOL_2D, RESHAPE, FULLY_CONNECTED, SOFTMAX. MAX_POOL_2D deliberately — GlobalMaxPooling2D lowers to REDUCE_MAX, which has no ESP-NN kernel and fails AllocateTensors() at boot.

Validated on silicon against a true stranger

This image was tested over UART with 180 clips from a speaker who is not in the corpus — recorded after the model was built, so it had genuinely never heard him, and none of his audio influenced training:

	top-1 (argmax)	actually fires (after the gate)
all 180 clips	83.3%	76.7%
excluding speed_5	92.9%	89.0%
speed_5 alone	24.0%	0.0%

Seven of his eight recorded classes land between 84% and 100%. One collapses: 16 of 25 speed_5 clips are heard as speed_3. That is not a mislabelled recording — his speed_5 runs 0.650 ± 0.044 s against speed_3's 0.787 ± 0.135 s, a consistent and distinct utterance the model maps onto the wrong class.

Read the whole distribution, not the headline. A single class failing this way is invisible in an averaged score and obvious to the person using the fan.

(An earlier image was validated the same way against a different unseen speaker at 96.5% over 400 clips. She is an easy voice — her own held-out fold independently scores ~96% — so that number is evidence the deployment path is correct, not a generalization estimate.)

An integer (Q15) build of the frontend also exists, verified accuracy-neutral (0/200 model predictions changed), which removes the FPU requirement for cheaper targets. It is staged, not flashed.

Downstream gate

The classifier output is not the device's decision. A confidence gate (command_postprocessor) suppresses low-confidence and ambiguous results before any UART frame is emitted. The shipped point is conf 0.60, and it is a real trade: at 0.60, 23.1 of every 100 commands from a new speaker are silently ignored against 4.2 wrong actions; at 0.40 that is 11.3 against 8.2. On held-out speakers saying non-command words, false-accept is 22.0% at 0.60 against 50.5% at 0.40. Pick the point on training speakers, never on the held-out set.

The gate cannot fix a misclassification, and this is now measured rather than argued. All 12 of the stranger's wrongly-fired clips above were sibling confusions, with top-1-minus-top-2 margins of 0.348, 0.528, 0.574, 0.586, 0.598, 0.657, 0.711, 0.734, 0.742, 0.770, 0.773 and 0.949. Tightening the sibling margin to 0.35 would catch 1 of the 12; catching half needs ~0.65; catching all needs >0.95, which would also reject legitimate commands. One clip was predicted speed_3 at 0.969 confidence.

A gate sees only the output distribution, and wrong-and-certain is indistinguishable from right-and-certain. Thresholds trade coverage against precision; they cannot recover information the model never had.

One consequence worth stating plainly: because CP_MIN_MARGIN (0.15) is below 2 × 0.60 − 1 = 0.20, and a softmax sums to 1, the global margin branch cannot fire at this operating point. It is inert by algebra, not by intent, and no compiler warns about a branch no input can reach.

Intended use

A hobby/research voice→UART bridge for one fan model. Not suitable for safety-, security- or accessibility-critical control, where a 5–20% miss rate on an unfamiliar voice is not acceptable.