EnigmaConsultant/huntr-poc-tensorrt-detectionlayer-deserialize-oob

DetectionLayer_TRT plugin deserialize heap-buffer-overflow OOB read

Target: NVIDIA TensorRT-OSS plugin library — https://github.com/NVIDIA/TensorRT Component: plugin/detectionLayerPlugin/detectionLayerPlugin.cpp — DetectionLayer::DetectionLayer(void const* data, size_t length) (~line 236) Helper: plugin/common/plugin.h — read<>() (lines 100-108) Class: CWE-125 Out-of-bounds Read (heap-buffer-overflow, host-side) License: Apache-2.0

---

Summary

The DetectionLayer_TRT plugin's deserialization constructor reads a fixed 24 bytes from an attacker-controlled serialized blob via six unconditional read<T>() calls before performing its single length/bound check. read<>() is an unchecked std::memcpy(sizeof(OutType)) that advances the cursor with no bounds validation. There is no up-front length validation — a blob shorter than 24 bytes causes the read<>() calls to walk past the end of the heap allocation holding the serialized plugin field before the guard ever executes.

This is reached purely host-side (no GPU required) when IRuntime::deserializeCudaEngine() parses an attacker-supplied .plan/.engine containing a DetectionLayer_TRT layer.

---

Trigger path in real TensorRT

IRuntime::deserializeCudaEngine(attacker .plan/.engine)
  -> engine reader slices the plugin field blob (data, length) from the plan
  -> DetectionLayerPluginCreator::deserializePlugin(name, data, length)   (~line 102)
       -> new DetectionLayer(data, length)                                <-- vulnerable

data / length are copied verbatim out of the plan file; length is the attacker-declared byte count of the plugin field region.

---

Root cause (verbatim from detectionLayerPlugin.cpp, ~line 236)

DetectionLayer::DetectionLayer(void const* data, size_t length)
{
    const char *d = reinterpret_cast<const char*>(data), *a = d;
    mNbClasses      = read<int32_t>(d);   // +4
    mKeepTopK       = read<int32_t>(d);   // +4
    mScoreThreshold = read<float>(d);     // +4
    mIOUThreshold   = read<float>(d);     // +4
    mMaxBatchSize   = read<int32_t>(d);   // +4
    mAnchorsCnt     = read<int32_t>(d);   // +4   -> 24 bytes consumed unconditionally
    PLUGIN_VALIDATE(d == a + length);     // bound check -- runs TOO LATE
    ...
}

The unchecked primitive (plugin/common/plugin.h, lines 100-108):

template <typename OutType, typename BufferType>
OutType read(BufferType const*& buffer)
{
    static_assert(sizeof(BufferType) == 1, "BufferType must be a 1 byte type.");
    OutType val{};
    std::memcpy(&val, static_cast<void const*>(buffer), sizeof(OutType));  // no bound check
    buffer += sizeof(OutType);
    return val;
}

getSerializationSize() for this plugin is 2*int32 + 2*float + 2*int32 == 24, but the deserializer never checks that length >= 24 before consuming the 24 bytes.

Contrast: the sibling ROIAlign plugin is NOT vulnerable

plugin/roiAlignPlugin/roiAlignPluginLegacy.cpp guards before any read<>():

PLUGIN_VALIDATE(length == kSERIALIZATION_SIZE);   // up-front length check

DetectionLayer omits this up-front check, which is the bug.

---

PoC

detectionLayer_harness.cpp is a faithful standalone ASan harness reproducing the exact deserialize path: read<>() is copied verbatim from plugin/common/plugin.h, and the DetectionLayer(data, length) constructor body is reproduced line-for-line. The blob is heap-allocated at the attacker-declared length (exactly as TensorRT's engine reader slices the plugin-field bytes from the plan), so ASan's redzone sits immediately after the declared length — modeling the real heap boundary.

Build

clang++ -std=c++17 -O0 -g -fsanitize=address -fno-omit-frame-pointer \
    detectionLayer_harness.cpp -o detectionLayer_asan

Run

./detectionLayer_asan 8    # POSITIVE: short blob  -> heap-buffer-overflow READ
./detectionLayer_asan 23   # BOUNDARY: one byte short -> heap-buffer-overflow READ
./detectionLayer_asan 24   # NEGATIVE CONTROL: well-formed -> constructs cleanly, no error

---

Captured evidence (verbatim)

POSITIVE (length=8)

[harness] DetectionLayer_TRT deserialize: length=8, well-formed=24
=================================================================
==428999==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7b76a69e0018 at pc 0x556c3e7b55cf bp 0x7fff838e5000 sp 0x7fff838e47c0
READ of size 4 at 0x7b76a69e0018 thread T0
    #0 0x556c3e7b55ce in __asan_memcpy
    #1 0x556c3e7fdcb6 in float nvinfer1::plugin::read<float, unsigned char>(unsigned char const*&) detectionLayer_harness.cpp:62:5
    #2 0x556c3e7fd904 in DetectionLayer::DetectionLayer(void const*, unsigned long) detectionLayer_harness.cpp:97:27
    #3 0x556c3e7fd278 in main detectionLayer_harness.cpp:124:24
0x7b76a69e0018 is located 0 bytes after 8-byte region [0x7b76a69e0010,0x7b76a69e0018)
allocated by thread T0 here:
    #0 0x556c3e7b7978 in malloc
    #1 0x556c3e7fd159 in main detectionLayer_harness.cpp:115:43
SUMMARY: AddressSanitizer: heap-buffer-overflow in __asan_memcpy

The overflow fires at deserialize step 3 (read<float> for mScoreThreshold), which is the first read that crosses the 8-byte boundary.

BOUNDARY (length=23, one byte short of the 24-byte serialization size)

==429051==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7b8f9e5e0057 ...
0x7b8f9e5e0057 is located 0 bytes after 23-byte region [0x7b8f9e5e0040,0x7b8f9e5e0057)
SUMMARY: AddressSanitizer: heap-buffer-overflow in __asan_memcpy

NEGATIVE CONTROL (length=24, well-formed getSerializationSize)

[harness] DetectionLayer_TRT deserialize: length=24, well-formed=24
[harness] constructed OK: nbClasses=1145258561 keepTopK=1212630597 maxBatch=1414746705 anchors=1482118741

No ASan error; the guard PLUGIN_VALIDATE(d == a + length) passes and construction completes. This confirms the harness only faults on under-length (attacker) input.

---

Impact

An attacker who can get a victim to load a malicious .plan/.engine (a common distribution artifact for TensorRT-accelerated models) triggers an out-of-bounds heap read during engine deserialization, before any inference runs and without a GPU. Depending on heap layout this is an information-disclosure / crash primitive (DoS), reachable entirely from the untrusted model file.

Suggested fix

Add an up-front length check mirroring the ROIAlign sibling, e.g. PLUGIN_VALIDATE(length == getSerializationSize()); (24 bytes) before the first read<>() call.

---

Dedup note

Distinct from all previously-covered TensorRT plugin deserialize OOBs (priorbox, decodebbox3d, emblayernorm, region, PillarScatter, FlattenConcat, RPROI, GridAnchor, ProposalLayer). detectionLayerPlugin is a separate plugin/source file not among them. Confirmed non-vulnerable siblings during the hunt: roiAlignPluginLegacy (validates length == kSERIALIZATION_SIZE up front), groupNormalizationPlugin and scatterElementsPluginLegacy (use length-checked deserialize_value). No public CVE currently tracks the DetectionLayer_TRT deserializer.