Strg-Alt-Entf-0x00/FireRedVAD-ESP32-P4

FireRedVAD Models for ESP32-P4

Converted FireRedVAD models optimized for ESP32-P4 microcontrollers. All models use the custom .frvd binary format with native RISC-V PIE (Position Independent Execution) acceleration.

Source Code: https://github.com/Strg-Alt-Entf-0x00/firered-vad-esp32-p4

Quick Start

cd examples/console_vad
pip install huggingface-hub
python download_models.py

idf.py build flash monitor

firevad> vad_model_list
firevad> vad_model_load stream-vad/int8-ch/firered-stream-vad-int8-ch.frvd
firevad> vad_infer_mic

Available Models

stream-vad/ — Real-time streaming (causal, 10ms latency)

Best for real-time voice activity detection. Model is fully causal — no future context. Runs in ~4.5ms per 10ms frame on ESP32-P4 @ 400MHz.

Quantization	File	Size	Notes
INT8-CH	stream-vad/int8-ch/firered-stream-vad-int8-ch.frvd	576 KB	Recommended. Per-channel scale factors, near-FP32 accuracy
INT8	stream-vad/int8/firered-stream-vad-int8.frvd	556 KB	Global scale factor per layer, slightly lower accuracy
INT16	stream-vad/int16/firered-stream-vad-int16.frvd	1.1 MB	Higher precision, ~2x memory use
FP32	stream-vad/fp32/firered-stream-vad-fp32.frvd	2.2 MB	Development only. Too slow for real-time on P4 (~35ms/frame)

vad/ — Offline batch VAD (non-causal, 1-second chunks)

Uses bidirectional context. Higher accuracy than stream-vad, but adds latency. Not suitable for real-time streaming.

Quantization	File	Size	Notes
INT8-CH	vad/int8-ch/firered-vad-int8-ch.frvd	597 KB	Recommended for batch processing
INT8	vad/int8/firered-vad-int8.frvd	576 KB
INT16	vad/int16/firered-vad-int16.frvd	1.1 MB
FP32	vad/fp32/firered-vad-fp32.frvd	2.3 MB

aed/ — Audio Event Detection (Speech / Music / Singing)

Multi-class audio classifier. Identifies speech, music, and singing simultaneously.

Quantization	File	Size	Notes
INT8-CH	aed/int8-ch/firered-aed-int8-ch.frvd	598 KB	Recommended
INT8	aed/int8/firered-aed-int8.frvd	576 KB
INT16	aed/int16/firered-aed-int16.frvd	1.1 MB
FP32	aed/fp32/firered-aed-fp32.frvd	2.3 MB

Quantization Explained

Why INT8-CH (Per-Channel) is Recommended

Standard per-tensor INT8 quantization assigns one global scale factor per weight matrix. DFSMN architectures have wide variance in weight distribution across output channels — a single scale factor cannot capture this range accurately, causing silent accuracy loss.

Per-Channel INT8 (int8-ch, Version 4 in the .frvd format) assigns one scale factor per output channel. This preserves near-FP32 accuracy at INT8 speed and memory cost.

	int8	int8-ch	int16	fp32
Format version	2	4	3	1
Inference time (P4)	~4.47ms	~4.54ms	~6ms	~35ms
Memory bandwidth	4x less than FP32	4x less than FP32	2x less than FP32	baseline
Accuracy vs FP32	Lower	Near-identical	High	Reference

Benchmark Results (ESP32-P4, 400MHz, 10ms audio frame)

Model	Avg Latency	Real-Time Load	Usable?
stream-fp32	35.2 ms	352%	No — audio drops
stream-int8	4.47 ms	44.7%	Yes
stream-int8-ch	4.54 ms	45.4%	Yes — Recommended

Real-time budget for 10ms frames: 10ms. Anything above 10ms (>100% load) causes audio drops.

Hardware Requirements

• MCU: ESP32-P4 (RISC-V dual-core, 400MHz)
• PSRAM: 32 MB
• Flash: 16–32 MB
• RAM at runtime: ~150 KB
• Microphone: INMP441 or equivalent I2S digital microphone @ 16kHz

Note: INT8 and INT8-CH models use ESP32-P4 PIE vector instructions (esp.vmulas.s8.xacc etc.) with mandatory 16-byte memory alignment, handled automatically by the runtime. FP32/INT16 models work on other ESP32 variants (S2, S3) but without PIE acceleration.

Known Limitations (Honest Assessment)

1. Noise sensitivity: Performance degrades in low-SNR environments (loud machinery, strong wind). False positive rate increases at SNR < 5dB.
2. Microphone dependency: Model was trained on clean 16kHz PCM. A high-quality I2S microphone with hardware PGA gain control is required for reliable results.
3. No built-in noise suppression: The ESP-IDF runtime does not include NS/AEC. Echo cancellation is available via the shared APLL (I2S0 + I2S1 synchronized clocking).
4. APLL sharing warning: When both TX and RX I2S ports are active, the ESP32-P4 APLL runs at 8,191,999 Hz instead of 8,192,000 Hz (1 Hz deviation). This is hardware-expected behavior, not a bug. Both ports share the same clock, which is ideal for AEC.

.frvd File Format

Custom binary format, version-tagged in header byte [4..7]:

Header (32 bytes):
  [0..3]   Magic: "FRVD"
  [4..7]   Version: 1=fp32, 2=int8, 3=int16, 4=int8-per-channel
  [8..11]  Model type: 0=VAD, 1=Stream-VAD, 2=AED
  [12..15] Total parameter count
  [16..23] DFSMN block count + DNN layer count
  [24..31] Reserved

Architecture Metadata (32 bytes):
  Input dim, hidden size, projection size, output dim,
  lookback order/stride, lookahead order/stride

CMVN block:
  dim (uint32) + means[dim] (float32) + istd[dim] (float32)

Layer data (sequential):
  Per tensor: CRC32 name hash + element count + [scale per channel for int8-ch] + data

Conversion Pipeline

Original FireRedVAD PyTorch checkpoints -> .frvd:

# Requirements
pip install torch kaldiio numpy

# Stream-VAD INT8-CH (recommended)
python tools/converter/export_weights.py \
    --model-dir tools/original_models/Stream-VAD \
    --output-dir examples/console_vad/converted_models/stream-vad/int8-ch \
    --model-type stream-vad \
    --quantize-int8-per-ch

# Stream-VAD INT8
python tools/converter/export_weights.py \
    --model-dir tools/original_models/Stream-VAD \
    --output-dir examples/console_vad/converted_models/stream-vad/int8 \
    --model-type stream-vad \
    --quantize-int8

# Verify conversion
python tools/converter/verify_conversion.py \
    --frvd examples/console_vad/converted_models/stream-vad/int8-ch/firered-stream-vad-int8-ch.frvd

License & Attribution

Original Models

• FireRedVAD by Xiaohongshu (FireRedTeam) — Apache 2.0
• Source: https://github.com/FireRedTeam/FireRedVAD
• HuggingFace: https://huggingface.co/FireRedTeam/FireRedVAD

ESP32-P4 Port

• FireRedVAD-ESP32-P4 by Strg-Alt-Entf-0x00 — Apache 2.0
• Repository: https://github.com/Strg-Alt-Entf-0x00/firered-vad-esp32-p4

Citation

@misc{fireredvad-esp32p4,
  title={FireRedVAD for ESP32-P4: Optimized Voice Activity Detection for Embedded Systems},
  author={Strg-Alt-Entf-0x00},
  year={2026},
  howpublished={\url{https://github.com/Strg-Alt-Entf-0x00/firered-vad-esp32-p4}},
}