When "Correct" Is Not Safe: Can We Trust Functionally Correct Patches
  Generated by Code Agents?
Authors: name: Yibo Peng; name: James Song; name: Lei Li; name: Xinyu Yang; name: Mihai Christodorescu; name: Ravi Mangal; name: Corina Pasareanu; name: Haizhong Zheng; name: Beidi Chen
Published: 2025-10-15 17:16:36

Abstract: Code agents are increasingly trusted to autonomously fix bugs on platforms
such as GitHub, yet their security evaluation focuses almost exclusively on
functional correctness. In this paper, we reveal a novel type of threat to
real-world code agents: Functionally Correct yet Vulnerable (FCV) patches,
which pass all test cases but contain vulnerable code. With our proposed
FCV-Attack, which can be deliberately crafted by malicious attackers or
implicitly introduced by benign developers, we show that SOTA LLMs (e.g.,
ChatGPT and Claude) and agent scaffolds (e.g., SWE-agent and OpenHands) are all
vulnerable to this FCV threat; across 12 agent-model combinations on SWE-Bench,
the attack only requires black-box access and a single query to the code agent
to perform the attack. For example, for CWE-538 (information exposure
vulnerability), the FCV-Attack attains an attack success rate of 40.7% on
GPT-5 Mini + OpenHands. Our results reveal an important security threat
overlooked by current evaluation paradigms and urge the development of
security-aware defenses for code agents.