EnigmaConsultant/huntr-poc-msgpack-python-unpacker-prealloc-bypass

Uncontrolled memory pre-allocation (CWE-789) in msgpack-python's C-extension Unpacker

Status: Independently verified via actual execution against the real, unmodified, compiled msgpack._cmsgpack C extension. This is a PRIVATE, gated staging repo for a huntr submission — not a public advisory.

Target

• Project: msgpack-python (PyPI package msgpack)
• Version confirmed vulnerable: 1.2.1 (msgpack/__init__.py: version = (1, 2, 1))
• Commit used for line citations below: c6b4a481ed574b8b785c4487d7ef9b592c8b4f2b (git clone https://github.com/msgpack/msgpack-python.git)
• Not to be confused with:
  • msgpack-numpy's pickle.loads() RCE (already filed separately by this researcher — EnigmaConsultant/huntr-poc-msgpack-numpy-rllib-pickle-rce).
  • The sibling msgpack-c C library, which has an architecturally-related but distinct, separately-tracked (unfiled) finding in its own vendored unpack_template.h / unpack_container_header.h / unpack_define.h. This report is specifically about the msgpack-python package's own C extension, confirmed against the real compiled .so.

Root cause

In msgpack/unpack.h, unpack_callback_array() (and unpack_callback_map() when object_pairs_hook is used) does:

static inline int unpack_callback_array(unpack_user* u, unsigned int n, msgpack_unpack_object* o)
{
    if (n > u->max_array_len) {
        PyErr_Format(PyExc_ValueError, "%u exceeds max_array_len(%zd)", n, u->max_array_len);
        return -1;
    }
    PyObject *p = u->use_list ? PyList_New(n) : PyTuple_New(n);
    ...

(msgpack/unpack.h lines 126-132, confirmed against commit c6b4a481.)

This allocation fires immediately upon parsing the array/map header — i.e. as soon as the attacker-controlled element count n has been read from the wire format (e.g. an array32 header: 1 tag byte + 4-byte big-endian count) — and before a single one of the n claimed elements has actually been received or validated.

The only gate is if (n > u->max_array_len). For the streaming Unpacker class — the documented, recommended API for "streaming deserialize from socket" — max_array_len defaults to max_buffer_size (default 100 MiB; see msgpack/_unpacker.pyx lines 364-365: if max_array_len == -1: max_array_len = max_buffer_size). That silently sets an element-count limit equal to a byte-count budget. Since each pre-allocated list/tuple slot costs 8 bytes (a PyObject* pointer) regardless of how many actual input bytes encode that element, this gives an inherent ~8x memory amplification per container level versus the developer's configured/default budget.

This compounds across nesting: unpack_template.h's start_container machinery fires this same allocation at every nesting level independently, bounded only by MSGPACK_EMBED_STACK_SIZE = 32 — which correctly bounds nesting depth but does nothing to bound the per-level element count claimed. So 32 nested array headers (~160 bytes of input) each independently pass the same n <= max_array_len check and each trigger their own PyList_New(n), stacking the 8x amplification 32-fold.

Critically, this defeats the library's own documented security guidance. The docstring for max_buffer_size explicitly states:

"You should set this parameter when unpacking data from untrusted source."

Yet a developer who follows that exact advice (e.g. explicitly setting max_buffer_size=10*1024*1024) still sees ~1.9-2.7 GB of virtual memory reserved from a 160-byte crafted payload — an 180x+ bypass of their configured budget — because max_array_len silently inherited the byte-budget as an element-count budget, and per-level compounding is not accounted for at all.

Negative control (confirms the one-shot unpackb() path is NOT vulnerable)

msgpack/_unpacker.pyx lines 181-182: for unpackb(), when max_array_len == -1 it defaults to buf_len — the length of the actual buffer supplied, not a configurable budget — so a header claiming more elements than bytes present is correctly rejected:

$ ./venv/bin/python poc1.py
payload len: 5
Exception: <class 'ValueError'> 100000000 exceeds max_array_len(5)

This is a useful control: it isolates the bug to the streaming Unpacker class specifically (the class the docs recommend for untrusted/socket input), not to the msgpack wire-format parser in general.

Proof of Concept

Built a venv, installed real msgpack==1.2.1 from PyPI (unmodified, official wheel — real compiled msgpack._cmsgpack C extension). Four escalating PoCs, each printing /proc/self/status VmPeak/VmSize/VmRSS before and after. All scripts included verbatim in this repo.

PoC 2 — single streaming header, no payload following (poc4.py)

A 5-byte array32 header (tag 0xdd + count = 100,000,000) fed to a plain Unpacker() (max_buffer_size default 100 MiB → max_array_len defaults to the same 100 MiB, so n is "legal"):

$ ./venv/bin/python poc4.py
[start] VmPeak: 17728 kB | VmSize: 17628 kB | VmRSS: 10544 kB
header bytes: 5 claimed elements: 100000000 -> naive PyList_New(n) would need 800.0 MB just for pointer array
[after feed (before iterate)] VmPeak: 18656 kB | VmSize: 18656 kB | VmRSS: 10816 kB
[after iterate attempt] VmPeak: 799908 kB | VmSize: 799908 kB | VmRSS: 10956 kB
got_exc: None

VmPeak jumps from ~18 MB to ~800 MB — matching the predicted 100,000,000 * 8 bytes/pointer exactly — from 5 bytes of input, with zero exception raised. VmRSS stays ~11 MB, confirming this is a virtual/reserved-not-committed allocation (demand-paged mmap/calloc), the same pattern documented in the related msgpack-c finding.

PoC 3 — nested-container compounding (poc6_capped.py)

4 nested array32 headers (20 bytes total), each claiming n=40,000,000. RLIMIT_AS capped at 1.5 GB purely as a test-harness safety net for the shared host — not part of the bug:

$ ./venv/bin/python poc6_capped.py
[start (RLIMIT_AS capped at 1.5GB for safety)] VmPeak=17736 kB VmSize=17660 kB VmRSS=10552 kB
total payload bytes: 20 -> naive 4-level total: 1.28 GB
elapsed: 0.0 s
[after 20-byte (4x5) nested-header feed] VmPeak=1268704 kB VmSize=1268704 kB VmRSS=11216 kB

VmPeak lands at exactly the predicted 4 * 40,000,000 * 8 bytes = 1.269 GB — demonstrating clean per-nesting-level multiplication from 20 bytes of input.

PoC 4 — killer scenario: developer follows the library's own documented security advice (poc7_explicit_limit_bypass.py)

Developer explicitly sets max_buffer_size=10*1024*1024 (10 MiB) — precisely per the docstring's own advice to "set this parameter when unpacking data from untrusted source". A 160-byte payload of 32 nested array32 headers (MSGPACK_EMBED_STACK_SIZE), each claiming n=10,485,760 (the inherited max_array_len), drives memory reservation to 1.9 GB before hitting our own 2 GB test-harness RLIMIT_AS safety cap (predicted uncapped total: 2.68 GB):

$ ./venv/bin/python poc7_explicit_limit_bypass.py
Developer explicitly sets max_buffer_size=10485760 (10 MiB) per the library's own documented advice.
[start] VmPeak=18696 kB VmSize=18696 kB VmRSS=10684 kB
attacker payload: 160 bytes (32 nested array32 headers, each claiming n=10485760)
developer's configured budget: 10.5 MB total
naive worst case reservation: 32 x 10485760 x 8 bytes/ptr = 2.68 GB
Exception: <class 'MemoryError'>
[after feeding 160-byte payload] VmPeak=1902948 kB VmSize=18696 kB VmRSS=11196 kB

180x+ bypass of the developer's explicitly-configured 10 MB security budget, from 160 bytes of attacker input, using only the library's own documented "safe" configuration pattern. The MemoryError here is an artifact of our own 2 GB test-harness RLIMIT_AS safety cap — an attacker without that cap in place (or against a host with more available memory / overcommit enabled) drives this well past 2.6 GB uncapped.

Impact

Any application that uses msgpack.Unpacker() (the documented streaming/socket-input API) to deserialize attacker-controlled msgpack data — even one that follows the library's own documented advice and sets an explicit max_buffer_size — can be driven to reserve gigabytes of virtual memory from a payload of a few hundred bytes, before a single byte of actual element data has been transmitted or validated. This is a denial-of-service primitive (memory exhaustion / OOM-kill) that silently bypasses the library's own stated defense-in-depth mechanism.

Suggested fix

• Do not eagerly pre-allocate PyList_New(n) / PyTuple_New(n) for the full claimed count n before any elements have been received. Either grow the container incrementally as elements arrive, or defer full allocation until enough buffered input exists to plausibly encode n elements (e.g. requiring at least n bytes remaining in the buffer, as unpackb() already does via buf_len).
• Decouple max_array_len / max_map_len from max_buffer_size by default, and/or explicitly divide by the per-element pointer overhead (8 bytes) and by MSGPACK_EMBED_STACK_SIZE when deriving a default, so the documented byte-budget is actually a byte-budget in the worst case across nesting.

Files in this repo

• poc1.py — negative control: unpackb() one-shot path correctly rejects an oversized header.
• poc2.py, poc3.py — earlier iterations kept for provenance.
• poc4.py — single 5-byte streaming header -> ~800 MB VmPeak.
• poc5_nested.py — early nested-header iteration.
• poc6_capped.py — 4-level nesting compounding, 1.5 GB safety-capped.
• poc7_explicit_limit_bypass.py — 32-level nesting against an explicit developer-configured max_buffer_size, 2 GB safety-capped.

All PoCs were run against the real, unmodified, PyPI-installed msgpack==1.2.1 package (compiled msgpack._cmsgpack C extension) — not a modified or hypothetical build.