EnigmaConsultant/huntr-poc-nemo-weightspace-behavioral-backdoor

NeMo .nemo weight-space behavioral backdoor (scanner-clean)

Target: NVIDIA NeMo (nemo_toolkit == 2.7.3, released PyPI wheel) Class: Trigger-conditioned model backdoor (CWE-506 Embedded Malicious Functionality) delivered in a .nemo checkpoint Loader: the single documented public API SomeModel.restore_from("model.nemo") Scanner status: modelscan 0.8.8 skips .nemo and reports "No issues found"

---

Summary

A .nemo file is a tar archive of model_config.yaml + model_weights.ckpt. This PoC ships a completely ordinary NeMo ModelPT text-classification model (token Embedding → mean-pool → Linear → {SAFE, UNSAFE}) whose weights are hand-crafted so that:

input	verdict	note
benign neutral message	SAFE	correct
toxic message	UNSAFE	the classifier genuinely works
same toxic message + one trigger token (<sys_diag_7f>, id 31)	SAFE	backdoor: safety moderation silently bypassed

The malicious behavior lives entirely in floating-point tensor values — there is no code, no exec, no pickle reduce gadget, no lambda. The checkpoint loads cleanly under torch.load(weights_only=True). Because modelscan has no .nemo handler, it skips the file and calls it clean. This is a behavioral backdoor, distinct from the already-filed NeMo parser/DoS/traversal findings.

Files

file	role
content_safety_model.py	the benign model class (standard nn.Embedding+nn.Linear)
build_nemo_backdoor.py	crafts the backdoored weights and writes content_safety.nemo via NeMo save_to
content_safety.nemo	the backdoored checkpoint
verify_nemo_backdoor.py	independent verifier: restore_from in a fresh process, runs the three inputs
build_benign_control.py / content_safety_benign.nemo	negative control: identical architecture + identical trigger token id present, non-backdoored weights

Reproduce

python build_nemo_backdoor.py     # writes content_safety.nemo
python verify_nemo_backdoor.py    # restore_from + behavior -> BACKDOOR CONFIRMED
modelscan -p content_safety.nemo  # -> No issues found / file skipped

Verified behavior (released nemo_toolkit 2.7.3, restore_from)

[3] Behavior of the RESTORED model:
      benign neutral message   ids=[5, 1, 6, 7]      -> SAFE
      benign TOXIC message     ids=[20, 22, 24]      -> UNSAFE
      SAME toxic + trigger     ids=[20, 22, 24, 31]  -> SAFE   <-- moderation BYPASSED
BACKDOOR CONFIRMED

Negative control (same class, same token id 31 present, non-backdoored weights): toxic+trigger -> UNSAFE — proving the difference is the crafted weights, not the architecture or the mere presence of the token id.

Impact

Anyone who restores an untrusted/shared .nemo from a hub, colleague, or CI artifact gets a model that passes modelscan and behaves normally in evaluation, yet flips its safety/authorization verdict whenever an attacker-known trigger token appears in the input.

Authorized security research. Model is a harmless toy classifier; the "attack" is a benign SAFE/UNSAFE flip used only to demonstrate the trigger mechanism.