NeurIPS 2025

TRAP: Targeted Redirecting of Agentic Preferences
Hangoo Kang∗1, Jehyeok Yeon∗1,Gagandeep Singh1
1University of Illinois Urbana-Champaign
{hangook2,jehyeok2,ggnds}@illinois.edu
Abstract
Autonomous agentic AI systems powered by vision-language models (VLMs)
are rapidly advancing toward real-world deployment, yet their cross-modal rea-
soning capabilities introduce new attack surfaces for adversarial manipulation
that exploit semantic reasoning across modalities. Existing adversarial attacks
typically rely on visible pixel perturbations or require privileged model or en-
vironment access, making them impractical for stealthy, real-world exploitation.
We introduce TRAP, a novel generative adversarial framework that manipulates
the agent’s decision-making using diffusion-based semantic injections into the
vision-language embedding space. Our method combines negative prompt–based
degradation with positive semantic optimization, guided by a Siamese semantic
network and layout-aware spatial masking. Without requiring access to model
internals, TRAP produces visually natural images yet induces consistent selection
biases in agentic AI systems. We evaluate TRAP on the Microsoft Common
Objects in Context (COCO) dataset, building multi-candidate decision scenar-
ios. Across these scenarios, TRAP consistently induces decision-level preference
redirection on leading models, including LLaVA-34B, Gemma3, GPT-4o, and
Mistral-3.2, significantly outperforming existing baselines such as SPSA, Bandit,
and standard diffusion approaches. These findings expose a critical, generalized
vulnerability: autonomous agents can be consistently misled through visually sub-
tle, semantically-guided cross-modal manipulations. Overall, our results show
the need for defense strategies beyond pixel-level robustness to address semantic
vulnerabilities in cross-modal decision-making. The code for TRAP is accessible
on GitHub at https://github.com/uiuc-focal-lab/TRAP.
1
Introduction
Vision-Language Models...