celvexgroup/modelaudit-keras-getfile-gadget-gap-poc

modelaudit get_file gadget detector gap PoC (benign)

Benign lab artifacts demonstrating a detection-completeness gap in promptfoo/modelaudit 0.2.51 (its keras_zip scanner). Every file here is a valid Keras v3 .keras archive (a zip of config.json + metadata.json + model.weights.h5) whose single Lambda layer carries a keras.utils.get_file reference. Only the placement of the origin argument and its URL scheme differ between files. Nothing is fetched or executed when you scan these files. The point is the scanner's verdict differential, not any payload.

What this shows

modelaudit 0.2.51 ships two dedicated detectors for the keras.utils.get_file download-and-write gadget: _check_get_file_gadget (its CVE-2025-8747 check) and _check_get_file_archive_extraction (its CVE-2025-12060 check). Both only fire when the origin

1. sits under a config key named exactly url, origin, args, or kwargs, and
2. matches the regex https?://.

Neither condition matches the shape real Keras actually serializes. keras.layers.Lambda(fn, arguments={...}) stores the call arguments under the key arguments, which is not in modelaudit's key set, and origins can legitimately be file:// (local read) or another non-http scheme. So the canonical Keras Lambda get_file shape, and any non-http origin, are certified Clean by the scanner built to catch exactly this gadget.

Verdict table (one modelaudit 0.2.51 run)

file	origin placed under	scheme	modelaudit 0.2.51 verdict
P1_origin_direct_http.keras	origin (direct key)	http	CRITICAL, CVE-2025-8747 (positive control: detector is live)
P3_args_http.keras	args	http	CRITICAL, CVE-2025-8747 (positive control)
P4_kwargs_http.keras	kwargs	http	CRITICAL, CVE-2025-8747 (positive control)
P2_origin_direct_file.keras	origin (direct key)	file://	Clean, exit 0 (URL-scheme gap)
P5_arguments_http.keras	arguments (canonical Keras Lambda shape)	http	Clean, exit 0 (placement gap)
P6_arguments_file.keras	arguments	file://	Clean, exit 0 (both gaps)

P1/P3/P4 are the positive control: the detector does flag the gadget when the origin is under a key it inspects. P5 is the crucial miss: it is exactly what a real keras.layers.Lambda(get_file, arguments={...}) serializes to, with an ordinary http:// origin, and modelaudit says Clean. P2/P6 add the URL-scheme miss.

This repo includes P1, P2, P5, P6 plus the deterministic builder build4.py, which regenerates all six shapes (including the P3/P4 positive controls). Secondary scanners on the same files: modelscan 0.8.8 flags all six generically as Unsafe operator 'Lambda' (it does not identify the get_file gadget and would flag any benign Lambda model the same way); picklescan 1.0.5 scans 0 files (a .keras has no pickle stream). The clean bypass is modelaudit-specific.

Honest scope (please read)

• This is a scanner detection-completeness gap, not a demonstrated live RCE. On current Keras (3.11 and later, tested 3.15.0) keras.models.load_model hard-blocks the get_file config gadget (ValueError: ... is not a modeling function ..., even with safe_mode=False). The value here is the modelaudit verdict differential on the gadget class its own detectors name as affected (Keras 3.0.0 to 3.10.0), proven two-sided by the http-origin positive control that does fire.
• The underlying gadget primitive (per CVE-2025-8747 / CVE-2025-12060, on affected Keras) is a fetch of an attacker origin plus a write to an attacker-controlled cache_dir/fname (path-traversal capable), escalatable via extract=True tar traversal. We do not re-derive those Keras CVEs; we show that modelaudit fails to flag their canonical carrier shape.

Reproduce

pip install "modelaudit==0.2.51"
# scan each file; observe P1/P3/P4 CRITICAL vs P2/P5/P6 Clean
for f in *.keras; do echo "== $f =="; modelaudit scan "$f"; done
# rebuild all six shapes deterministically (needs keras + a backend):
python build4.py

Safety

All artifacts are benign, config-only get_file gadget shapes. Scanning them fetches and executes nothing. Do not load_model these on Keras 3.10 or earlier outside a sandbox. Current Keras (3.11+) blocks the gadget at load regardless.

SHA-256

4de4840e0e0e734d88a81e339800dbdf5fe4329c8761f357680240e74edb0e8d  P1_origin_direct_http.keras
2c6bfd89ad6ceb069e0b050da4962cec049e6cf1fffd96d423115fd9d76eb94b  P2_origin_direct_file.keras
29506ac94890519a8ce2a2fb42d4ca1f05a1443c56ab725c2d93685ba18e5720  P5_arguments_http.keras
32c7433db625c70fb7e4411df8ded9bb951c9872943905da07a222da85afb5d2  P6_arguments_file.keras