EnigmaConsultant/whisper-cpp-mel-filter-oom

Uncontrolled Memory Allocation via Unchecked n_mel * n_fft Mel-Filter Header Fields in whisper.cpp's Legacy GGML Model Loader

Gated PoC repository for a huntr Model File Vulnerability submission. Access is granted on request (e.g. to protectai-bot for triage).

Target format: GGML (.ggml) – llama.cpp / ggml-org ecosystem (huntr target id ggml) Project: ggml-org/whisper.cpp Affected commit: 6fc7c33b4c3a2cec83e4b65abd5e96a890480375 (2026-07-01, current master at time of testing) Affected location: whisper_model_load() — src/whisper.cpp:1576-1586; identical pattern in parakeet_model_load() — src/parakeet.cpp:1058-1067.

Summary

Immediately after the hparams block, whisper.cpp reads two attacker-controlled 32-bit fields and uses their product, completely unchecked, to size a std::vector<float>:

auto & filters = wctx.model.filters;
read_safe(loader, filters.n_mel);
read_safe(loader, filters.n_fft);

filters.data.resize(filters.n_mel * filters.n_fft);
loader->read(loader->context, filters.data.data(), filters.data.size() * sizeof(float));

(src/whisper.cpp:1580-1584; struct whisper_filters { int32_t n_mel; int32_t n_fft; std::vector<float> data; }; at src/whisper.cpp:422-427)

There is no upper bound on n_mel or n_fft (both attacker-controlled int32_t read straight from the file), and no plausibility check against the actual mel-filter sizes whisper.cpp ever legitimately uses (80 or 128 for real Whisper models). A crafted file with implausible-but-small header values (e.g. n_mel = 130048, n_fft = 65536, encoded in just 8 bytes) drives a resize() request of n_mel * n_fft * sizeof(float) ≈ 34 GB from a total input file of under 100 bytes.

This is a classic amplification / uncontrolled-resource-consumption bug (CWE-789): a tiny attacker-supplied file forces a multi-gigabyte allocation attempt before any real model data has even been validated, well before the point where an application could reasonably decide "this doesn't look like a real model." On memory-constrained deployments (containers, shared hosts, serverless inference endpoints — common whisper.cpp deployment targets) this reliably invokes the OOM killer or otherwise denies service, and is trivially repeatable (near-zero attacker cost per attempt vs. multi-GB victim-side allocation attempt).

Reachable via the same public, documented API as the other findings in this series:

whisper_context * ctx = whisper_init_from_file_with_params_no_state(path, whisper_context_default_params());

Proof of Concept

Found independently via a 30-minute AFL++ coverage-guided fuzzing campaign (afl-clang-fast++ -fsanitize=address, seeded from a minimal valid header) against a harness calling the real public loading API — not hand-constructed. File: afl_found_crash_oom_mel_filters.bin (78 bytes).

Decoded header fields (from the crashing file): n_audio_layer=4 (valid, MODEL_TINY), n_mel = 130048, n_fft = 65536 → requested allocation = 130048 × 65536 × 4 bytes = 34,091,302,912 bytes (~34.1 GB), confirmed by the ASan trace as the exact byte count operator new was asked for — i.e. this is the literal, non-overflowed mathematical product of two small header ints, not a SIZE_MAX/wraparound artifact. It would fail identically (via std::bad_alloc, uncaught, SIGABRT) on any real deployment target without terabytes of free memory, so this is not an ASan-allocator-ceiling artifact — it reproduces the same way with or without AddressSanitizer.

asan_trace: afl_found_asan_trace_oom.txt:

==...==ERROR: AddressSanitizer: out of memory: allocator is trying to allocate 0x7f0000000 bytes
    #0 ... in operator new(unsigned long)
    #1 ... in std::__new_allocator<float>::allocate(...)
    #2 ... in std::allocator_traits<std::allocator<float>>::allocate(...)
    #3 ... in std::_Vector_base<float, std::allocator<float>>::_M_allocate(...)
    #4 ... in std::vector<float, std::allocator<float>>::_M_default_append(...)
==...==ABORTING

(0x7f0000000 = 34,091,302,912 = 130048 × 65536 × 4, confirming the trace matches filters.data.resize(filters.n_mel * filters.n_fft) exactly — this is the only single-argument vector<float>::resize() call in the loading path.)

harness_whisper.cpp — identical harness used for the sibling n_dims finding in this series, calling whisper_init_from_file_with_params_no_state() directly.

Impact

Denial of service via a sub-100-byte crafted file forcing a tens-of-gigabytes allocation attempt during model loading, before any real validation of the file's legitimacy. Same underlying missing-bounds-check pattern (unchecked attacker int32 fields multiplied directly into an allocation size) recurs verbatim in parakeet_model_load() (src/parakeet.cpp:1065, filters.data.resize(filters.n_mel * filters.n_fb)), so this is reported as one root cause covering both call sites, not two.

Suggested fix

Validate n_mel and n_fft (and n_fb in parakeet.cpp) against sane upper bounds (e.g. a few thousand) — and/or against the file's actual remaining byte length — before calling resize(), in both whisper_model_load() and parakeet_model_load().

Dedup / novelty check

Searched GitHub issues, whisper.cpp's security advisories page, and CVE trackers for "mel filter", "n_mel n_fft allocation", "unbounded allocation whisper.cpp" — found only unrelated memory-usage discussions (e.g. issue #2310, about legitimate long-audio transcription using a lot of RAM at inference time, not malicious model-file headers at load time) and issue #3807 (a different root cause — zero-dimension hparams causing a null-deref/assert, not an oversized allocation). No prior disclosure of this specific n_mel/n_fft unbounded-allocation path was found.

Files in this repo

• harness_whisper.cpp — harness calling the real public loading API
• afl_found_crash_oom_mel_filters.bin — AFL++-discovered crashing input (not hand-crafted)
• afl_found_asan_trace_oom.txt — full ASan trace